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PREFACE 


The  Los  Angeles  County  Board  of  Supervisors  (the  Board)  adopted  the 
Long-Term  Family  Self-Sufficiency  (LTFSS)  Plan  on  November  16,  1999.  The 
LTFSS  Plan  consists  of  46  projects  whose  goal  is  to  promote  self-sufficiency 
among  families  that  are  participating  in  the  California  Work  Opportunity  and 
Responsibility  to  Kids  Act  of  1997  (CalWORKs)  program,  former  CalWORKs 
families,  and  other  low-income  families.  The  Chief  Administrative  Office  (CAO) 
is  the  lead  agency  responsible  for  implementing  the  evaluation  of  the  LTFSS 
Plan.  On  December  5,  2000,  the  Board  approved  the  implementation  plan  for 
the  evaluation  of  the  LTFSS  Plan,  LTFSS  Project  #46.  Following  an  open  and 
competitive  bidding  process,  the  Board  awarded  RAND  a  contract  to  conduct  a 
Countywide  evaluation  of  the  LTFSS  Plan. 

The  LTFSS  Plan  is  explicitly  guided  by  Mark  Friedman’s  Results-Based 
Decision  Making  (RBDM)  Framework  (the  RBDM  Framework).  LTFSS  Plan 
experiences  with  the  RBDM  Framework  can  provide  early  indications  of  the  utility 
of  the  RBDM  Framework,  the  challenges  that  are  likely  to  be  encountered  as 
County  departments  attempt  to  apply  the  RBDM  Framework  in  the  County,  and 
approaches  to  addressing  those  challenges.  This  document  is  intended  to 
extract  those  lessons  learned  by  providing  insights  into  the  use  and  utility  of  the 
RBDM  Framework  as  it  has  been  adopted  and  adapted  by  the  County.  Most  of 
the  analysis  in  this  report  represents  work  completed  as  of  the  end  of  October 
2001. 

For  more  information  about  RAND’s  evaluation  of  the  LTFSS  Plan,  contact: 


Elaine  Reardon 

Project  Director  and  Associate  Economist 
RAND 

1700  Main  Street 
Santa  Monica,  CA  90407 
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EXECUTIVE  SUMMARY 


INTRODUCTION 

The  California  Work  Opportunity  and  Responsibility  to  Kids  Act  of  1997 
(CalWORKs)  provided  Los  Angeles  County  (hereafter,  the  County)  with  a  large 
new  stream  of  funding  (over  $150  million  per  year  for  five  years)  with  relatively 
few  conditions.  On  April  13,  1999,  the  County  Board  of  Supervisors  (hereafter, 
the  Board)  instructed  the  County’s  New  Directions  Task  Force  (NDTF)  to  develop 
a  Long-Term  Family  Self-Sufficiency  (LTFSS)  Plan  for  CalWORKs  recipients  and 
the  broader  low-income  population,  with  the  purpose  of  selecting  projects  that 
would  improve  the  capacity  of  families  to  become  self-sufficient.  The  NDTF 
effort  resulted  in  the  46  projects  known  collectively  as  the  LTFSS  Plan  that  was 
approved  by  the  Board  on  November  16,1 999. 

In  developing  and  implementing  the  Plan,  the  NDTF  and  the  workgroups  it 
created  were  explicitly  guided  by  Mark  Friedman’s  Results-Based  Decision 
Making  (RBDM)  Framework  (hereafter,  the  RBDM  Framework).  The  use  of  the 
RBDM  Framework  has  three  important  implications  for  the  development  of  the 
Plan.  First,  the  RBDM  Framework  urges  a  focus  not  just  on  how  well  agencies 
and  projects  perform,  but  on  population-level  results  (i.e.,  the  well-being  of 
children,  families,  and  communities).  Consistent  with  this  focus  on  population- 
level  results,  the  LTFSS  Plan  projects  are  designed  to  address  five  key 
outcomes:  good  health,  safety  and  survival,  economic  well-being,  social  and 
emotional  well-being,  and  education  and  workforce  readiness. 

Second,  the  RBDM  Framework  urges  an  open  process,  emphasizing  the 
importance  of  opening  funding  deliberations  from  narrow  department  and  agency 
discussions  to  the  broader  community.  It  does  so  both  because  broad 
community  involvement  leads  to  decisions  that  better  reflect  the  preferences  of 
the  population  and  because  having  an  impact  on  indicators  of  well-being  is 
viewed  as  a  collaborative  process  between  government,  community-based 
organizations  (CBOs),  and  individual  citizens. 
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Third,  the  RBDM  Framework  focuses  on  performance  accountability: 
Projects  are  to  track  project-level  performance  measures  which  quantify  service 
inputs  and  client  outcomes.  The  results  of  this  tracking  are  expected  to  flow  into 
future  project  funding  decisions.  Countywide  indicators  of  population-level 
outcomes  should  also  be  tracked,  and  this  information  should  feed  back  into 
decisions  by  the  community  about  how  available  funds  should  be  allocated. 

The  County  has  now  embarked  on  an  ambitious  effort  to  use  the  RBDM 
Framework  throughout  the  County’s  planning  and  budgeting  process  for  child 
and  family  well-being  programs.  LTFSS  Plan  experiences  with  the  RBDM 
Framework  can  provide  early  indications  of  the  utility  of  the  RBDM  Framework, 
the  challenges  that  are  likely  to  be  encountered  as  County  departments  attempt 
to  apply  the  RBDM  Framework  in  the  County,  and  approaches  to  addressing 
those  challenges.  This  document  is  intended  to  extract  those  lessons  learned  by 
providing  insights  into  the  use  and  utility  of  the  RBDM  Framework  as  it  has  been 
adopted  and  adapted  by  the  County. 

To  assess  the  utility  of  the  RBDM  Framework,  RAND  conducted  a  set  of 
semi-structured  interviews  with  key  informants.  In  total,  the  research  team 
interviewed  approximately  65  observers,  gathering  a  wide  range  of  perspectives. 
The  interviewees  included  senior  County  officials,  staff  in  individual  departments, 
and  non-governmental  observers  involved  in  developing  and  implementing  the 
LTFSS  Plan.  In  addition  to  these  semi-structured  interviews,  RAND  collected 
and  analyzed  a  wide  range  of  written  materials — official  actions  of  the  Board, 
internal  planning  documents,  budgets,  and  related  California  Department  of 
Social  Services  (CDSS)  documents.  These  materials  were  augmented  with 
broader  perspectives  from  other  ongoing  RAND  research  on  California  welfare 
policy  and  the  selection  and  implementation  of  social  programs.  Finally,  this 
evidence  was  interpreted  using  relevant  social  science  and  management 
research  drawn  from  the  fields  of  organizational  behavior,  economics,  and 
sociology. 
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THE  LTFSS  PLAN  EXPERIENCE  APPLYING  THE  FISCAL  POLICY  STUDIES 
INSTITUTE’S  RBDM  FRAMEWORK 

The  planning  process  that  led  to  the  development  of  the  LTFSS  Plan  was 
viewed  by  participants  as  representing  a  “first”  for  the  County  in  several  respects. 
The  RBDM  Framework  proved  to  be  a  useful  planning  tool  (though  as  we 
discuss  below,  its  application  could  be  improved).  The  planning  process  brought 
a  wider  array  of  perspectives  to  the  table  than  had  been  the  case  in  other 
planning  efforts.  Furthermore,  the  planning  process  helped  the  County  to  think 
more  broadly  about  the  goals  it  wanted  to  achieve.  The  fact  that  the  Department 
of  Public  Social  Services  (DPSS)  shared  the  available  funds  with  other  County 
departments  (i.e.,  that  these  monies  flowed  from  the  DPSS  to  other  lead 
agencies)  was  also  seen  as  an  important  step  forward  from  traditional  stove¬ 
piping  of  departmental  resources. 

Despite  the  fact  that  the  RBDM  Framework  has  relatively  little  to  say  about 
the  details  of  implementation,  the  Framework  urges  an  overarching  plan  focused 
on  achieving  population-level  outcomes  (such  as  family  self-sufficiency).  It 
states  that  achieving  the  desired  outcomes  is  the  responsibility  of  the  community, 
that  the  community  is  accountable  for  those  results.  This  broad  effort  requires 
partners  to  work  together,  implying  that  service  delivery  programs  will  entail 
community  involvement.  When  planning  moves  toward  these  goals,  the  resulting 
programs  tend  to  be  more  complicated  to  implement  than  conventional 
programs:  Any  attempt  to  break  down  long-established  ways  of  doing  business 
creates  implementation  challenges.  Therefore,  a  natural  question  in  assessing 
the  utility  of  the  RBDM  Framework  is  the  extent  to  which  the  use  of  the  RBDM 
Framework  affects  the  speed  of  implementation. 

As  of  October  2001 ,  two  and  a  half  years  after  the  initiation  of  the  planning 
process  and  two  years  after  the  approval  of  the  LTFSS  Plan,  startup  of  service 
delivery  to  clients  is  only  now  getting  under  way  for  a  number  of  projects.  About 
half  the  projects  have  approved  implementation  plans  (such  that  they  could 
begin  to  serve  clients),  and  some  projects  have  approved  evaluation 
deliverables.  For  most  projects,  the  provision  of  those  services  appears  to  be  at 
least  several  months  away.  This  schedule  is  clearly  slower  than  had  been 
expected. 
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In  general,  project  roll-out  has  gone  about  as  fast  as  should  have  been 
expected.  The  LTFSS  Plan’s  project  roll-out  has  been  slow  because  LTFSS 
projects  have  three  levels  of  challenges  to  confront,  many  of  which  derive  from 
the  use  of  the  RBDM  Framework  in  planning  the  initiative.  First,  even  though 
there  was  high-level  involvement  in  the  planning  of  the  LTFSS  projects,  rolling 
out  new  programs — using  a  RBDM  Framework  or  not — always  takes  a  long  time. 
Experience  with  other  programs  suggests  that  even  for  standard  programs,  this 
process  takes  9-12  months,  sometimes  more.  This  consideration  alone  would 
have  been  expected  to  delay  actual  service  delivery  until  late  2000  or  early  2001 . 

Second,  LTFSS  projects  are  often  not  standard  new  projects  or  simply 
expansions  of  existing  programs.  Each  of  the  fundamental  ideas  of  the  LTFSS 
Plan  and  the  RBDM  Framework— increasing  community  involvement,  flowing 
funds  between  departments,  and  service  integration— requires  that  a  number  of 
details  be  worked  out  about  cross-agency  cooperation  and  coordination  and 
collaboration  with  the  community.  Doing  so  would  be  expected  to  stretch  out  the 
initial  implementation  phase. 

Third,  because  the  LTFSS  Plan  was  a  prototype  for  a  new  model  of  service 
provision  in  the  County,  there  were  few  precedents  for  how  to  accomplish  service 
provision.  Before  the  details  of  cross-agency  coordination  for  each  individual 
project  could  be  worked  out,  structures  and  procedures  for  attending  to  those 
details  needed  to  be  developed. 

It  seems  reasonable  to  attribute  some  of  the  delay  in  project  roll-out  to  the 
use  of  the  RBDM  Framework  itself.  Some  of  the  delay  will  also  occur  with  future 
use  of  the  RBDM  Framework  (the  second  item  above,  the  coordination  issues). 
However,  if  the  lessons  from  the  LTFSS  Plan  experience  are  learned,  some  of 
the  delays  (the  third  item  above,  developing  procedures)  need  not  reoccur. 

Furthermore,  the  history  of  the  LTFSS  Plan  to  date  can  be  summarized  as 
follows:  the  County  has  shown  that  it  can  plan  and  implement  according  to  this 
new  model — planning  for  results,  as  conceptualized  by  the  RBDM  Framework. 
As  of  October  2001,  initial  procedures  have  been  developed  and  most  of  the 
steps  have  been  completed  for  at  least  some  of  the  projects.  As  experience 
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accumulates,  refined  procedures  and  processes  should  allow  for  improved 
application  of  the  RBDM  Framework. 

Those  involved  in  planning  and  implementing  the  LTFSS  Plan  effort  expect 
two  types  of  long-term  benefits  from  the  global  strategic  planning.  First, 
interviewees  feel  that  the  LTFSS  Plan  had  begun  moving  the  County  further 
along  toward  the  goal  of  service  integration. 

Second,  interviewees  see  a  number  of  potential  benefits  to  clients  when  all 
the  LTFSS  projects  have  begun  providing  services.  Projected  benefits  include 
more  services  to  families,  less  fragmentation  of  services  to  these  families, 
improvements  in  customer  service,  more  knowledgeable  line  staff  about  the 
range  of  County  programs  available  to  their  clients,  and  more  effective  referrals. 

KEY  LESSONS  LEARNED  IN  APPLYING  THE  RBDM  FRAMEWORK 

From  our  observation  of  the  County’s  first  experience  with  using  the  RBDM 
Framework  to  develop  and  implement  the  Plan,  we  draw  three  lessons. 

Performing  the  RBDM  Framework’s  Planning  Tasks  Requires  Significant 
Time 


Leaders  and  planning  participants  accomplished  a  great  deal  in  a  short 
planning  time  frame.  Through  a  collaborative,  consensus-building  process, 
those  involved  in  the  LTFSS  planning  process  agreed  on  a  definition  of  long-term 
family  self-sufficiency,  selected  a  set  of  Countywide  outcome  indicators  to 
measure  progress  toward  achieving  that  goal,  and  developed  a  suite  of  projects 
that  represented  the  “how”  of  improving  the  selected  outcome  indicators. 

That  said,  most  LTFSS  planning  leaders  and  participants  would  have  liked 
more  time  to  work  through  the  tasks  of  the  RBDM  Framework  and  to  fully 
consider  the  different  issues  that  arose.  A  key  lesson  learned  is  that  future 
planning  efforts  using  the  RBDM  Framework  would  benefit  by  having  more  than 
six  months  to  finalize  a  Plan  and  develop  projects.  More  time  would  allow 
planners  to  bring  the  best  available  evidence  to  the  table  in  determining  “what 
works”  and  what  outcome  indicators  should  be  used  to  measure  progress  toward 
achieving  the  agreed-upon  end  result(s)  or  outcome(s).  It  would  also  allow 
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planners  to  fully  involve  all  the  relevant  stakeholders  (e.g.,  community  advocacy 
groups,  researchers,  and  service  providers)  in  the  planning  process  and  to 
assess  what  sen/ices  already  exist  in  the  community  and  how  the  developed 
projects  will  relate  to  one  another  and  to  the  existing  service  delivery  systems. 
Finally,  having  more  time  would  allow  planners  to  develop  a  broader  consensus. 

In  Large  Counties,  More  Formal  Links  Between  Planning  and 
Implementation  Are  Needed 

In  a  large  county  such  as  Los  Angeles  County,  the  individuals  responsible 
for  planning  an  initiative  will  often  not  be  the  individuals  responsible  for 
implementing  an  initiative.  Thus,  those  implementing  may  not  understand  the 
planners’  intent.  Furthermore,  even  when  they  understand  the  planners’  intent, 
implementers  may  feel  that  the  program  is  not  feasible  or  does  not  represent  the 
best  use  of  department  resources.  Therefore,  planning  and  implementing  an 
initiative  the  magnitude  and  complexity  of  the  LTFSS  Plan  requires  addressing 
three  sets  of  issues. 

First,  it  is  essential  to  get  the  high-level  involvement  of  the  relevant  players 
and  of  community  representatives  in  the  planning  process  and  consensus¬ 
building  to  reach  an  agreed-upon  global  action  plan.  Second,  as  the  LTFSS 
Plan  moves  from  planning  to  implementation,  a  number  of  details  must  be 
worked  out  about  cross-agency  cooperation  and  coordination  (e.g.,  the  transfer 
of  funds  between  departments;  standardizing  reporting  procedures  or 
memorandum  of  understanding,  MOU,  formats;  determining  what  contracting 
requirements  will  be;  and  cross-supervision  of  departmental  staff).  Third,  the 
individual  lead  agency  and  program  staff  responsible  for  implementing  a  specific 
project  must  determine  what  program  infrastructure  (e.g.,  site  selection,  hiring 
and  training  of  staff,  recruitment  of  program  participants)  must  be  in  place  to  start 
up  sen/ice  delivery. 

To  successfully  address  these  three  issues,  it  would  help  if  the  Plan  had 
specified  more  formal  links  between  planning  and  implementation  (e.g., 
mechanisms  for  ensuring  the  inclusion  of  County  and  non-County  service 
providers  or  agency  representatives  in  the  planning  process  who  would  be 
responsible  for  implementing  an  initiative). 
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In  practice,  the  LTFSS  Plan  represents  a  change  from  “business  as  usual” 
in  the  County  and  the  innovations  of  the  RBDM  Framework  pose  some  inherent 
challenges  in  addressing  these  three  issues.  The  RBDM  Framework 
encourages  broad  community  input  at  planning  and  emphasizes  collaboration 
and  partnerships.  It  is  silent,  however,  about  how  this  should  be  accomplished. 
Actual  implementation  is  a  full-time  job,  of  necessity,  done  by  lead  agency 
staff— employees  of  a  particular  County  department  or  agency.  Thus,  one  of  the 
challenges  is  maintaining  community  involvement  and  ensuring  that  the  final 
projects  faithfully  implement  the  planners’  and  community’s  intent. 

To  ensure  the  continuity  of  the  vision  from  the  planning  process  to  the 
implementation  phase  and  to  maintain  the  focus  on  outcomes,  several 
participants  suggested  ongoing  Workgroup  meetings  that  involve  County  and 
non-County  representatives  who  participated  in  developing  the  LTFSS  Plan. 
Such  meetings  may  be  a  useful  forum  for  addressing  programmatic  issues  and 
constraints  that  planners  may  not  have  fully  considered  and  for  reconciling  them 
with  the  planners’  vision.  Meeting  invitees  might  include  Workgroup  planning 
participants,  lead  agency  and  program  representatives,  researchers  or  outside 
program  experts,  and  contract  and  finance  representatives. 

Because  the  dollars  for  LTFSS  projects  come  out  of  welfare  reform  and 
federal  Temporary  Assistance  for  Needy  Families  (TANF)  legislation,  transferring 
these  funds  from  one  department  to  other  departments  or  agencies  with  different 
funding  requirements  complicates  implementation.  New  procedures  needed  to 
be  put  in  place  and  additional  restrictions  were  placed  on  how  funds  could  be 
used.  In  retrospect,  senior  planners  would  have  involved  finance  and  contracts 
representatives  early  on  in  the  planning  phase.  Doing  so  would  allow  them  to 
begin  addressing  such  issues  as  departmental  fund  transfers,  and  establishing 
agreed-upon  reporting  procedures  and  document  templates  for  the 
implementation  phase.  Also,  interviewees  recognize  that  ongoing  technical 
assistance  is  valuable  to  project  staff  in  the  receiving  departments  or  agencies  to 
help  them  develop  project  implementation  plans  and  contracts  that  meet  CDSS 
and  federal  TANF  requirements. 

Another  lesson  learned  was  the  importance  of  recognizing  the  support 
needs  of  the  numerous  coordination  activities  that  must  occur  early  on  during  the 
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planning  phase  and  continue  into  the  implementation  phase.  As  noted  by 
several  senior  management  leaders,  the  LTFSS  Plan’s  budget  divides  a  project’s 
funds  evenly  across  the  five  years  and,  by  doing  so,  implicitly  assumes  rapid 
program  start-up  and  scale-up  to  full  operating  capacity.  However,  as  discussed 
above,  to  make  the  changes  from  “business  as  usual”  embodied  in  the  LTFSS 
Plan  and  to  undertake  the  innovations  of  the  RBDM  Framework,  a  number  of 
issues  related  to  cross-agency  cooperation  and  coordination  must  be  addressed 
early  on.  In  addition,  a  great  deal  must  occur  before  projects  are  ready  for 
service  delivery.  In  particular,  projects  must  contract  with  service  providers, 
select  program  sites,  hire  and  train  staff,  recruit  program  participants,  develop 
program  materials,  among  a  host  of  other  logistical  issues.  As  a  result, 
interviewees  felt  that  budgets  should  realistically  reflect  the  life  cycle  of  a  project, 
including  early  funding  to  address  the  numerous  coordination  issues  between 
agencies  prior  to  service  delivery,  then  funding  for  a  period  of  “ramping  up.” 

The  Evaluation  Component  Is  More  Complicated  Than  It  Appears  at  First 

Consistent  with  the  RBDM  Framework,  the  LTFSS  Plan  Evaluation  Design 
includes  a  two-level  evaluation.  At  the  population  level,  an  outside  evaluator — 
RAND — is  conducting  an  overall  assessment  of  the  County’s  progress  toward 
the  result,  self-sufficiency,  and  the  five  outcomes  by  measuring  progress  on  26 
indicators,  such  as  the  infant  mortality  and  the  poverty  rate,  and  by  assessing  the 
usefulness  of  the  RBDM  Framework  as  a  way  of  organizing  diverse  efforts 
toward  a  general  desired  result.  At  the  program  level,  each  of  the  LTFSS 
projects  is  required  to  measure  progress  on  performance  measures  to  determine 
its  own  success. 

Consistent  with  the  RBDM  Framework,  the  LTFSS  Plan’s  Evaluation 
Design  emphasizes  that  progress  on  selected  performance  measures  must  be 
measured.  Among  the  fundamental  insights  of  the  RBDM  Framework  is  the 
importance  of  refocusing  project  efforts  away  from  a  narrow  focus  on  what  the 
project  did  to  a  broader  focus  on  how  the  project  affected  client-level  outcomes 
and  well-being.  To  that  end,  the  RBDM  Framework  requires  that  each  project 
explicitly  state  which  Plan  outcomes  it  expects  to  affect,  operationalizing  those 
outcomes  through  specific  program  performance  measures  and  then  carefully 
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tracking  the  evolution  of  those  measures  over  time.  Individual  projects  are  urged 
to  track  whether  their  “customers  are  better  off  (Friedman,  2001 ,  Section  3.16). 

Building  an  evaluation  component  into  the  LTFSS  Plan  helps  to  keep  the 
key  stakeholders  and  lead  agencies  focused  on  outcomes  and  informs  the  Board 
about  how  these  funds  were  used.  The  Plan’s  evaluation  component  also 
introduces  County  departmental  staff  to  Friedman’s  approach  and  to  the  notion 
of  explicitly  thinking  about  what  a  program  is  trying  to  achieve  and  how  to  assess 
whether  what  an  agency  is  doing  is  having  the  desired  effects.  Participants  in 
the  planning  and  implementation  of  the  LTFSS  Plan  have  praised  the  application 
of  the  RBDM  Framework  as  having  guided  County  departments  and  other  lead 
agencies  in  thinking  more  formally  about  program  outcomes  and  as  providing  a 
useful  framework  for  prioritizing  departmental  resources. 

However,  some  interviewees  also  expressed  concern  about  whether  the 
Evaluation  Design  will  allow  for  the  determination  of  causation,  i.e.,  the  effect  of 
an  individual  LTFSS  project  (or  even  the  Plan  as  a  whole)  on  outcomes.  The 
RBDM  Framework  is  both  quite  specific  and  vague  about  how  to  evaluate  the 
effect  of  a  given  project  or  a  broad  effort.  The  specific  instructions  involve  using 
historical  information  to  create  a  trendline  (see  Friedman,  2001,  Section  2.11) 
and  then  using  the  historical  trend  information  to  project  a  future  trend  in  the 
absence  of  new  programs.  The  history  and  forecast  create  a  baseline.  Program 
success  is  evaluated  in  terms  of  deviations  between  baseline  and  actual 
outcomes.  If  historical  data  are  not  available,  the  RBDM  Framework  directs 
projects  to  collect  data  on  comparison  groups. 

However,  the  RBDM  Framework  notes  that  simple  pre-  versus  post¬ 
comparisons  are  invalid  because  other  non-project  forces  would  cause  outcomes 
to  change  even  in  the  absence  of  a  new  program.  The  examples  in  the  RBDM 
Framework  and  the  description  of  the  “Story  Behind  the  Baseline”  process 
emphasize  that  many  factors  will  affect  future  outcomes  (see  the  example  about 
“adequate  housing”  in  Section  2.11  of  Friedman,  2001)  and  that  not  all  the 
factors  are  captured  by  a  simple  trendline.  However,  neither  the  RBDM 
Framework  nor  the  Chief  Administrative  Office’s  (CAO’s)  technical  assistance 
materials  offers  statistical  methods  about  how  to  control  for  such  other  factors  in 
the  historical  projections  or  the  comparison  group  designs. 
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Those  performing  project  evaluations  would,  therefore,  benefit  from 
additional  training  and  technical  assistance.  Current  CAO  training  materials 
describe  the  application  of  the  basic  trendline  methods  suggested  by  Friedman. 
However,  the  recent  path  of  the  economy — rapid  expansion,  followed  by  at  least 
a  mild  recession — suggests  that  trendline  methods  are  likely  to  underestimate 
the  true  effects  of  the  program.  The  trendline— computed  during  the 
expansion — implicitly  assumes  that  the  expansion  will  continue.  Applying 
trendline  methods,  any  deviation  from  that  trendline — e.g.,  because  of  the  end  of 
the  expansion— will  be  attributed  to  the  program.  Thus,  for  some  outcomes, 
simple  trendline  analyses  are  likely  to  suggest  that  the  Plan  made  outcomes 
worse.  Alternative  methodologies  exist  that  will  (at  least  partially)  correct  for 
economic  conditions.  Thus,  it  may  be  useful  to  consider  significant  increases  in 
the  level  of  (and  funding  for)  training  for  those  performing  the  project  evaluations 
and  technical  assistance  in  developing,  refining,  and  implementing  their 
evaluation  plans  so  that  they  can  detect  any  positive  effects  of  the  Plan,  despite 
a  slowing  of  economic  growth. 

FUTURE  DIRECTIONS 

These  lessons  learned  become  a  useful  base  on  which  to  build  as  the 
County  moves  forward  both  to  extend  the  application  of  the  RBDM  Framework 
outside  the  LTFSS  Plan  and  to  continue  the  use  of  the  RBDM  Framework  within 
it. 


For  the  application  of  the  RBDM  Framework  outside  the  LTFSS  Plan,  a 
number  of  key  stakeholders  view  the  LTFSS  Plan  as  an  important  opportunity  to 
test  the  feasibility  of  applying  the  RBDM  Framework  in  the  County.  From 
experience  using  the  RBDM  Framework  in  developing  and  evaluating  the  LTFSS 
Plan,  the  County  can  infer  what  adjustments  may  be  needed  to  apply  the  RBDM 
Framework  to  other  areas  of  the  County’s  budget.  Such  learning  opportunities 
are  particularly  important  given  the  County’s  recent  decision  to  incorporate  the 
RBDM  Framework  into  the  County’s  budget  process,  specifically,  into  the 
Children’s  Budget. 

In  terms  of  the  continued  use  of  the  RBDM  Framework  in  the  LTFSS  Plan, 
the  implementation  of  the  LTFSS  Plan  is  moving  into  its  third  calendar  year,  and 
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lead  agencies  will  move  from  an  emphasis  on  developing  projects’ 
implementation  plans  and  putting  an  initial  program  in  place  to  an  emphasis  on 
service  delivery,  refining  LTFSS  projects,  overseeing  contractors,  and  evaluating 
these  projects  and  tracking  outcomes.  Fortunately,  the  lead  agencies  and  DPSS 
and  CAO/SIB  begin  work  on  these  tasks  from  a  solid  base. 

In  its  use  of  the  RBDM  Framework  in  the  LTFSS  Plan  and  in  its  extension 
of  that  application  to  other  parts  of  the  County  budget,  the  County  is  ahead  of 
other  jurisdictions  in  applying  the  basic  concept  of  “managing  for  results”  to  best 
serve  the  needs  of  children  and  low-income  families.  Further,  by  incorporating 
an  evaluation  component  into  the  Plan,  the  County  is  ahead  of  others  in  its  quest 
to  learn  and  improve  on  the  RBDM  Framework’s  application.  The  utility  of  the 
RBDM  Framework  as  an  evaluation  framework  remains  to  be  assessed  as 
projects  proceed  with  their  evaluations.  Regardless,  clearly  the  preliminary 
lessons  learned  from  this  “experiment”  will  help  inform  future  planning  and 
application  of  the  RBDM  Framework  within  the  County. 


This  Page  Intentionally 
Left  Blank 


-  XXIII  - 


ACKNOWLEDGMENTS 


This  report  has  greatly  benefited  from  the  contributions  of  a  number  of 
individuals.  We  wish  to  acknowledge  the  participants  in  the  Countywide  planning 
process,  lead  agency  staff,  and  non-County  representatives  who  willingly  shared 
their  experiences  in  planning  and  implementing  the  LTFSS  Plan.  We  also 
express  our  thanks  to  those  who  provided  us  with  access  to  planning 
documents,  expenditure  data,  and  other  information  related  to  the  Plan’s 
development  and  implementation. 

We  also  thank  the  LTFSS  project  coordinators,  the  Long-Term  Family  Self- 
Sufficiency  Division  and  Financial  Management  Division,  DPSS,  and  CAO/SIB 
staff  for  their  assistance  in  ensuring  we  had  access  to  needed  data  and  other 
information  in  support  of  our  fieldwork. 

Finally,  we  wish  to  express  our  appreciation  to  Robert  Schoeni,  Amy  Cox, 
and  Jeffrey  Wasserman  for  their  thoughtful  review  and  comments  on  the  draft 
report.  Christopher  Dirks  and  Patrice  Lester  provided  valuable  assistance  in 
preparing  this  report. 


This  Page  Intentionally 
Left  Blank 


XXV  - 


ACRONYMS 


AFDC 

Aid  to  Families  with  Dependent  Children 
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LEP 

Limited  English  Proficient 

LTFSS 

Long-Term  Family  Self-Sufficiency 

MOU 

Memorandum  of  Understanding 
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Act  of  1996 
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1.  INTRODUCTION 


BACKGROUND 

The  California  Work  Opportunity  and  Responsibility  to  Kids  Act  of  1997 
(CalWORKs)  provided  Los  Angeles  County  (hereafter,  the  County)  with  a  large 
new  stream  of  funding  (over  $150  million  per  year  for  five  years)  with  relatively 
few  conditions.  On  April  13,  1999,  the  County  Board  of  Supervisors  (hereafter, 
the  Board)  instructed  the  County’s  New  Directions  Task  Force  (NDTF)  to  develop 
a  Long-Term  Family  Self-Sufficiency  (LTFSS)  Plan  for  CalWORKs  recipients  and 
the  broader  low-income  population,  with  the  purpose  of  selecting  projects  that 
would  improve  the  capacity  of  families  to  become  self-sufficient.  The  NDTF 
effort  resulted  in  the  46  projects  known  collectively  as  the  LTFSS  Plan  that  was 
approved  by  the  Board  on  November  16, 1999. 

The  influence  of  Fiscal  Policy  Studies  Institute’s  Results-Based  Decision 
Making  Framework  (hereafter,  the  RBDM  Framework)  on  the  Long-Term  Family 
Self-Sufficiency  Plan  (hereafter,  the  LTFSS  Plan)  is  apparent  at  many  levels— in 
the  process  used  to  choose  projects,  in  the  projects  actually  chosen,  and  in  the 
requirements  that  each  project  measure  their  performance.  As  we  will  see  in  the 
body  of  the  report,  these  RBDM  Framework-based  ideas  have  had  important 
consequences  for  the  projects  chosen  and  for  the  speed  with  which  those 
projects  have  begun  providing  services  to  County  citizens. 

The  County  has  now  embarked  on  an  ambitious  effort  to  use  the  ideas  of 
the  RBDM  Framework  throughout  the  County’s  planning  and  budgeting  process 
for  child  and  family  well-being  programs.  LTFSS  Plan  experiences  with  the 
RBDM  Framework  can  provide  early  indications  of  the  utility  of  the  RBDM 
Framework  and  of  the  challenges  that  are  likely  to  be  encountered  as  County 
departments  attempt  to  apply  the  RBDM  Framework  in  the  County. 

OBJECTIVE 

As  part  of  an  ongoing  contract  with  the  County’s  Chief  Administrative  Office 
(CAO),  RAND  is  conducting  the  Countywide  evaluation  of  the  LTFSS  Plan.  That 
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evaluation  will  produce  three  annual  reports,  each  of  which  will  be  comprised  of 
the  results  of  both  an  impact  analysis  and  a  process  analysis. 

The  impact  analysis  component  is  intended  to  identify  and  track  measures 
of  outcomes  and  the  effects  of  the  LTFSS  Plan.  As  we  discuss  in  detail  in  the 
next  chapter,  these  activities— analyzing  the  population-level  outcomes  that  the 
County  would  like  to  affect  in  reaching  its  goal  of  family  self-sufficiency — are 
themselves  key  components  of  the  RBDM  Framework. 

To  date,  RAND  has  released  two  documents  as  part  of  the  impact  analysis: 
(1)  a  report  that  evaluated  the  data  sources  that  can  be  used  to  measure  the  26 
indicators  identified  in  the  LTFSS  Plan  and  that  identified  five  “headline 
indicators”  from  the  original  26  that  will  be  used  to  track  each  of  the  five 
proposed  outcome  areas  (Hedderson  and  Schoeni,  2002);  and  (2)  a  report  that 
lays  out  a  “baseline”  trend  for  the  headline  indicators  and  that  describes  the 
factors  that  have  caused  the  baseline  trends  to  appear  as  they  do  for  each  of  the 
headline  indicators — i.e.,  telling  the  “story  behind  the  baseline”  (Schoeni  et  al., 
2002). 

The  process  analysis  component  is  intended  to  provide  complementary 
insights  into  the  use  and  utility  of  the  RBDM  Framework  as  it  has  been  adopted 
and  adapted  by  the  County.  As  we  noted  earlier,  the  development  and 
evaluation  of  the  LTFSS  Plan  were  strongly  influenced  by  the  RBDM  Framework. 
As  the  County  moves  to  expand  the  use  of  the  RBDM  Framework,  experiences 
with  using  the  RBDM  Framework  in  designing  and  evaluating  the  LTFSS  Plan 
and  its  component  projects  can  provide  important  insights  and  lessons  learned. 

APPROACH 

In  general  terms,  process  analyses  generate  constructive  suggestions  for 
improving  processes  and  procedures  by  combining  three  types  of  information: 

•  A  careful  and  rich  description  of  experiences  to  date.  What  are 
the  official  processes  and  procedures?  What  processes  and 
procedures  were  actually  used?  To  what  extent  have  the 
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processes  and  procedures  succeeded  in  generating  the  desired 
outcomes? 

•  The  impressions  and  suggestions  of  participants  and  observers 
in  the  LTFSS  Plan.  What  has  “worked”?  And,  what  has  not 
“worked”?  Which  people,  organizations,  policies,  and 
procedures  have  helped  and  which  have  hindered?  And,  most 
important,  how  may  problems  be  resolved? 

•  The  project  team’s  observations  of  similar  programs  in  other 
evaluations  and  perspectives  from  relevant  academic  literatures. 

In  general,  the  first  two  types  of  information  are  collected  through  project- 
specific  field  work  and  document  analysis,  while  the  third  type  of  information  is 
drawn  from  the  previous  experiences  and  training  of  the  project  staff.  Combining 
these  three  types  of  information  yields  suggestions  for  changes  in  policies  and 
procedures,  in  this  case,  suggestions  related  to  the  RBDM  Framework. 

At  the  core  of  this  research  strategy  is  the  collection  of  detailed  information 
on  actual  implementation  to  date  and  the  perspectives  of  participants  and 
observers  involved  in  planning  or  implementing  the  LTFSS  Plan.  To  collect  such 
information,  RAND  designed  and  implemented  a  multi-method  research  plan. 
The  foundation  of  the  research  plan  is  a  set  of  semi-structured  interviews  with 
key  informants.  In  preparation  for  the  semi-structured  interviews,  RAND 
developed  a  set  of  research  questions  and  working  hypotheses.  These  research 
questions  and  working  hypotheses  were  then  used  to  develop  a  master  interview 
protocol. 

With  the  assistance  of  the  CAO  and  the  Department  of  Public  Social 
Services  (DPSS),  RAND  identified  knowledgeable  potential  interviewees  and 
assigned  particular  questions  from  the  master  interview  protocol  to  specific 
interviewees.  Most  interviews  were  conducted  with  two-person  teams.  Some 
interviews  were  taped,  transcribed,  and  added  to  the  project’s  computer  files;  for 
the  other  interviews,  detailed  notes  were  taken,  transferred  to  computer  files, 
reviewed  by  project  staff,  and  added  to  the  project  files. 

In  total,  the  research  team  interviewed  approximately  65  observers, 
gathering  a  wide  range  of  perspectives.  The  interviewees  included  the  following: 
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•  County  departmental  staff  from  each  of  the  lead  agencies 

•  DPSS  staff  responsible  for  planning  and  implementing  the 
LTFSS  Plan 

•  CAO/Services  Integration  Branch  (SIB)  staff  responsible  for  the 
evaluation  of  the  LTFSS  Plan 

•  Non-County  departmental  participants,  including  advocacy 
groups,  nonprofit  organizations,  and  academicians. 

To  encourage  frank  discussion  of  procedures  and  issues  and  following 
standard  procedure  in  such  efforts,  RAND  promised  anonymity  to  all 
interviewees.  Thus,  except  with  their  explicit  permission,  we  do  not  quote  them 
by  name  nor  even  list  their  names. 

In  addition  to  these  semi-structured  interviews,  RAND  collected  and 
analyzed  a  wide  range  of  written  materials,  which  included  the  following: 

•  Official  actions  of  the  Board 

•  Planning  documents,  as  well  as  related  memorandum  and 
correspondence 

•  Documents  of  the  Workgroups’  planning  activities,  including 
minutes  of  meetings,  notes  taken  by  individual  participants,  and 
related  correspondence 

•  Training  materials,  including  implementation  guides  and  other 
technical  guides  developed  for  use  by  the  lead  agencies  and 
Workgroup  participants 

•  Projects’  implementation  and  evaluation  deliverables,  as  well  as 
project  documentation 

•  Project  status  update  reports  and  other  written  materials 

•  Related  California  Department  of  Social  Services  (CDSS) 
documents. 

These  materials  were  augmented  with  broader  perspectives  from  other 
ongoing  RAND  research  on  California  welfare  policy  and  the  selection  and 
implementation  of  social  programs,  including  RAND’s  Statewide  Evaluation  of 
the  California  Work  Opportunity  and  Responsibility  to  Kids  (CalWORKs)  program 
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and  its  Evaluation  of  the  Comprehensive  Youth  Services  Act  (CYSA).  Both 
evaluations  examine  the  impact  of  welfare-to-work  (WTW)  reform  within 
California  and  the  use  of  federal  Temporary  Aid  to  Needy  Families  (TANF) 
funding  by  the  State  of  California. 

Finally,  this  evidence  was  interpreted  using  relevant  social  science  and 
management  research  drawn  from  the  fields  of  organizational  behavior, 
economics,  and  sociology. 

ORGANIZATION  OF  THIS  DOCUMENT 

The  balance  of  this  report  proceeds  in  five  chapters.  Chapter  2 — Overview 
of  the  RBDM  Framework  and  the  LTFSS  Plan  Process — describes  the  RBDM 
Framework  and  provides  an  overview  and  status  of  the  planning, 
implementation,  and  service  delivery  processes.  Chapter  3 — Findings  About  the 
Utility  of  the  RBDM  Framework  in  the  Planning  Process — discusses  the  utility  of 
the  RBDM  Framework  in  planning  the  LTFSS  Plan.  Chapter  A — Findings  About 
the  Effects  of  the  LTFSS  Plan  on  the  Implementation  Process — discusses  the 
service  delivery  approach  taken  by  the  LTFSS  Plan.  Chapter  5 — Findings  About 
the  Utility  of  the  RBDM  Framework  in  the  Evaluation  Process— discusses  the 
utility  of  the  RBDM  Framework  in  the  evaluation  process. 

Finally,  Chapter  6 — Lessons  Learned  and  Future  Directions — concludes 
this  report  with  a  summary  of  the  key  lessons  learned  and  a  discussion  of  future 
directions  in  applying  the  RBDM  Framework. 


This  Page  Intentionally 
Left  Blank 


-7- 


2.  OVERVIEW  OF  THE  RBDM  FRAMEWORK  AND  THE  LTFSS  PLAN 

PROCESS 


INTRODUCTION 

As  we  noted  in  Chapter  1 ,  the  goal  of  this  report  is  to  explore  the  use  and 
utility  of  the  RBDM  Framework  in  developing  and  implementing  the  LTFSS  Plan. 
To  understand  our  findings  on  the  application  and  utility  of  the  RBDM 
Framework— which  appear  in  Chapters  3,  4,  and  5 — it  is  crucial  to  first  provide 
some  context  for  those  findings  and  to  provide  an  overview  of  the  LTFSS  Plan 
effort. 


Thus,  in  this  chapter,  we  begin  with  an  overview  of  the  RBDM  Framework 
and  the  LTFSS  Plan,  as  well  as  of  the  Los  Angeles  Children’s  Planning  Council 
(CPC),  which  served  as  a  bridge  between  the  two.  We  then  outline  the  steps 
with  which  the  LTFSS  Plan  was  put  into  place,  from  the  initial  planning  phase 
through  to  the  implementation  and  evaluation  phases.  Our  discussion  considers 
the  time  frame  over  which  this  occurred,  as  well  as  the  policies  and  procedures 
that  evolved  to  create  a  plan  and  then  to  turn  a  plan  on  paper  into  an  actual 
enterprise  that  could  improve  families’  well-being.  Finally,  we  report  on  the 
status  of  these  efforts  as  of  the  end  of  October  2001 .  In  subsequent  chapters, 
we  assess  the  utility  of  the  RBDM  Framework  as  it  was  adopted  and  adapted  by 
the  County,  based  in  large  part  on  the  County’s  experience  as  described  in  this 
chapter. 

THE  RBDM  FRAMEWORK 

Like  the  body  of  this  report,  we  organize  our  overview  of  the  RBDM 
Framework  into  three  parts:  planning,  implementation,  and  evaluation.  Here,  we 
provide  a  high-level  discussion  of  each  of  these  parts. 

Planning 

In  the  County,  as  in  most  governments,  planning  occurs  within  limited 
domains.  To  deliver  sen/ices,  governments  are  organized  into  functional 
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departments.  In  the  County,  such  functional  departments  include  DPSS,  which 
is  responsible  for  CalWORKs,  Food  Stamps,  and  MediCal;  the  Department  of 
Health  Services  (DHS),  which  is  responsible  for  health  programs;  the 
Department  of  Mental  Health  Services  (DMHS),  which  is  responsible  for  mental 
health  programs;  and  the  Probation  Department  (PROB),  which  is  responsible 
for  the  supervision  of  court-mandated  offenders,  both  adults  and  juveniles.  In 
addition,  there  are  also  County-related  but  separate  agencies,  such  as  the  Los 
Angeles  County  Office  of  Education,  which  is  responsible  for  providing  education 
services. 

This  organizational  structure,  while  necessary  for  providing  services,  has 
the  unintended  and  unfortunate  effect  of  encouraging  narrow  planning  within  the 
individual  organizations,  while  discouraging  coordinated  planning  across  multiple 
departments  and  agencies.  As  a  result,  program  consequences  tend  to  be 
narrowly  focused  on  the  domain  of  the  particular  department  or  agency,  ignoring 
interactions  with  other  programs  in  other  departments  and  agencies.  The 
resulting  service  delivery  structure  is  often  compartmentalized  and  difficult  for 
citizens  to  use.  Instead  of  providing  a  holistic  approach  with  integrated  service 
provision,  the  structure  addresses  each  service  issue  and  each  family  member 
separately.  This  tendency  is  reinforced  both  by  standard  bureaucratic 
considerations  (e.g.,  the  desire  of  senior  bureaucrats  to  protect  and  expand  their 
organizations)  and  by  categorical  external  funding.  Such  categorical  external 
funding  usually  flows  to  a  particular  department  or  agency  (e.g.,  the  designated 
welfare  agency)  and  may  be  used  only  for  specified  purposes.  Together,  these 
factors  discourage  both  global  planning  across  multiple  departments  and  the 
integrated  delivery  of  services. 

Another  feature  of  planning  is  that  performance  has  traditionally  been 
measured  in  terms  of  the  effort  expended  rather  than  on  ultimate  outcomes. 
How  many  cases  did  the  agency  serve?  What  was  the  response  time?  Were 
there  complaints?  These  narrow  measures  focus  on  the  efforts  of  individual 
programs  rather  than  on  “results” — i.e.,  effects  on  the  client-level  outcomes.  This 
approach  is  imperfect  because,  as  Friedman  says,  “trying  hard  is  not  good 
enough.  We  need  to  be  able  to  show  results  to  taxpayers  and  voters” 
(Friedman,  2001). 
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The  RBDM  Framework  is  intended  to  change  these  two  traditional  aspects 
of  government  planning  and  budgeting.  It  urges  that  planning  start  with  ends 
and  work  backwards  to  means  (Friedman,  2001).  Specifically,  before  selecting 
projects,  planners  should  identify  the  population-level  outcomes  they  want  their 
programs  to  affect.  These  outcomes  are  then  operationalized  through  the 
selection  of  specific  indicators. 

This  focus  on  population  outcomes  then  provides  an  overarching  objective 
for  planning  and  budgeting.  Rather  than  focusing  on  individual  funding  sources 
and  individual  programs  in  isolation,  the  RBDM  Framework  emphasizes 
population  outcomes,  encouraging  global  planning  across  programs  and  funding 
streams.  How  can  we  coordinate  services  provided  by  different  departments  to 
address  these  outcomes?  How  can  we  deliver  services  from  multiple 
departments  in  a  way  that  is  seamless  (or  at  least  less  fragmented  than  current 
practice)?  How  can  we  move  funds  coming  into  one  department  to  the 
department  that  can  operate  the  program  that  is  most  likely  to  affect  the 
outcomes  of  highest  priority?  These  indicators  are  to  assess  the  program's  (in 
this  case,  the  LTFSS  Plan's)  effect  on  outcomes.  To  do  so,  historical  data  on 
these  indicators  is  used  to  construct  a  forecast  of  future  levels  of  the  indicators  in 
the  absence  of  the  new  program.  Actual  outcomes  in  the  future  are  compared  to 
pre-program  outcomes  and  to  this  forecast  baseline  to  assess  the  effect  of  the 
program. 

In  the  RBDM  Framework,  the  selected  population  outcomes  and  their 
corresponding  indicators  are  used  to  select  individual  projects.  Projects  are  then 
to  develop  evaluations  to  assess  the  impact  of  their  efforts  on  client-level 
outcomes.  Specifically,  historical  data  on  the  outcomes  are  used  to  plot  the 
historical  trend  and  this  trend  is  then  used  to  develop  a  forecast  of  future  levels 
of  the  indicator  in  the  absence  of  program  intervention.  Together,  these  two 
trends  are  considered  a  baseline.  Underlying  this  baseline  is  a  “story  behind  the 
baseline”  that  incorporates  an  analysis  of  the  forces  that  shift  the  indicator.  This 
analysis  is  important  for  two  reasons.  First,  it  suggests  causal  paths  through 
which  programs  might  affect  the  indicator.  Second,  it  suggests  factors  beyond 
simple  extrapolation  of  recent  trends  that  need  to  be  considered  in  developing 
the  forecast. 


- 10- 


Finally,  the  RBDM  Framework  calls  on  projects  to  identify  “partners”— 
public-  and  private-sector  agencies  that  can  help  to  affect  the  indicators.  Such 
partners  are  particularly  important  in  the  RBDM  Framework  approach.  The 
RBDM  Framework  is  explicitly  and  deliberately  open.  It  emphasizes  the 
importance  of  opening  deliberations  from  narrow  department  and  agency 
discussions  to  the  broader  community.  It  does  so,  both  because  open 
deliberations  make  for  decisions  that  better  reflect  the  preferences  of  the 
population  and  because  changing  indicators  (“turning  the  cun/e”)  is  viewed  as  a 
collaborative  process  between  government,  community-based  organizations 
(CBOs),  and  individual  citizens. 

Thus,  in  the  planning  stage,  the  community  chooses  a  global  action  plan 
composed  of  individual  projects.  Ideally,  the  resulting  projects  are  very  different 
from  “business  as  usual.”  They  are  developed  with  broad  community  input, 
rather  than  from  within  individual  departments.  As  much  as  possible,  they 
transferred  funds  from  the  departments  nominally  receiving  the  monies  to  the 
departments  that  can  best  implement  the  programs  that  are  most  likely  to  affect 
the  population-level  outcomes.  Finally,  they  sometimes  involve  close 
coordination  and  coordination  or  integration  of  services  across  County 
departments,  both  at  the  senior  management/planning  level  and  at  the 
caseworker/service  delivery  level. 

Implementation 

The  RBDM  Framework  is  nearly  silent  about  implementation,  but  its  vision 
for  planning  implies  major  changes  in  the  programs  to  be  implemented  and  the 
challenge  of  doing  so.  Developing  new  programs  is  always  harder  than  running, 
or  even  refining,  existing  programs.  Furthermore,  line  staff  whose  primary 
responsibility  is  managing  ongoing  programs  may  have  little  experience  or 
training  in  developing  new  programs. 

This  inherent  challenge  of  rolling  out  new  programs  is  compounded  by  the 
innovations  of  the  RBDM  Framework.  While  actual  implementation  is  a  full-time 
job,  of  necessity,  done  by  employees  of  a  particular  County  department,  the 
RBDM  Framework  encourages  broad  community  input  at  planning.  Thus,  the 
challenge  is  to  maintain  community  involvement  and  to  try  to  ensure  that  the  final 
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project  faithfully  implements  community  intent.  In  addition,  the  Plan  moved 
funds  from  receiving  departments  or  agencies  to  those  that  can  best  implement 
the  programs.  Such  flowing  of  funds,  however,  complicates  implementation. 
Rather  than  one  responsible  department,  there  are  now  two  (or  more).  The 
effort  required  to  develop  programs  more  than  doubles.  It  is  not  enough  that 
each  department  review  and  approve  the  same  plan.  Implementation  cannot 
proceed  until  both  departments  approve  the  plans  and  such  approval  cannot 
occur  until  any  disagreements  about  the  plans  are  surfaced  and  resolved.  Doing 
so  requires  a  high  level  of  coordination  by  the  two  oversight  groups.  Delays 
because  of  coordination  and  priority  setting  should  be  expected. 

The  Plan  also  encourages  the  integration  of  the  delivery  of  services  at  the 
caseworker/service  delivery  level.  Doing  so,  in  some  instances,  requires  joint 
oversight  of  line  operations  and  the  interconnection  of  lines  of  control,  and  any 
decisions  made  by  front-line  staff  must  be  consistent  with  the  regulations  of  both 
departments.  There  also  needs  to  be  regular  communication  between  line  staff 
in  different  departments.  Developing  such  coordination  can  be  difficult  and  time- 
consuming. 

Accomplishing  all  this  would  be  difficult  enough  in  an  environment  in  which 
there  were  precedents  and  procedures  for  such  issues  (e.g.,  continuing 
community  input,  flowing  funds  across  departments,  coordinating  front-line 
service  delivery).  In  fact,  as  we  will  see,  the  LTFSS  Plan  represents  a  first 
attempt  along  many  of  these  dimensions.  As  a  result,  to  proceed  with  the 
implementation  of  specific  projects,  basic  procedures  and  processes  needed  to 
be  established. 

These  implementation  challenges  put  a  premium  on  careful  oversight  of 
the  implementation  process.  This  means  setting  specific  intermediate  goals  and 
deadlines  for  those  goals  and  then  carefully  tracking  process  toward  those 
intermediate  goals  and  the  timing  of  that  progress.  (See  Friedman,  2001 .) 

The  Plan’s  emphasis  on  service  integration  also  underscores  the 
importance  of  interdepartmental  coordination.  High-level  planning  for 
interdepartmental  coordination  is  relatively  easy.  Senior  management  can  make 
agreements  among  themselves.  There  is  often  attention  (and  implicit  pressure) 
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from  global  management  (in  the  case  of  the  County,  the  Board)  for  such 
coordination  and  cooperation.  However,  actual  implementation  of  such  cross¬ 
department  efforts  is  likely  to  be  more  difficult  because  it  takes  place  out  of  the 
spotlight  of  global  management  and  involves  a  group  of  people  much  larger  than 
just  senior  management.  Therefore,  a  key  issue  in  the  final  success  of  global 
planning  efforts  will  be  how  well  such  efforts  succeed  in  keeping  narrow 
department-specific  considerations  and  priorities  from  redirecting  or  even 
derailing  cross-departmental  initiatives. 

Evaluation 

The  final  component  of  the  RBDM  Framework  is  a  focus  on  evaluation 
through  a  particular  evaluation  model.  We  have  already  discussed  some  of  the 
components  of  the  RBDM  Framework  evaluation  model  in  our  discussion  of 
planning. 

LOS  ANGELES  CHILDREN’S  PLANNING  COUNCIL 

Locally,  these  ideas — outcome-based  planning,  service  integration, 
measurement  of  outcomes — and  the  RBDM  Framework  have  been  championed 
by  the  Los  Angeles  Children’s  Planning  Council  (CPC),  a  Countywide 
public/private  collaborative.  Originally  established  in  1991  by  the  Board,  with 
roots  going  back  another  decade,  the  CPC  (along  with  its  proponents)  had 
worked  to  craft  a  unified  vision  of  child  policy  within  the  County.  (For  more  on 
the  CPC,  see  McCroskey,  2001 .)  That  unified  vision  is  intended  “to  integrate  the 
fragmented  child  and  family  services  system,  support  the  development  of 
comprehensive  community-based  initiatives,  and  move  toward  collaborative 
cross-system,  cross-jurisdiction  planning”  (McCroskey,  2001).  Specifically,  the 
CPC  has  focused  on  outcomes  and  the  integrated  delivery  of  services  through  a 
series  of  publication  efforts,  including  developing  a  “Children’s  Scorecard”  for 
Los  Angeles  (2001,  1999,  1998,  1995,  1994,  1986),  defining  “service  planning 
areas”  (SPAs),  and  compiling  a  unified  “Children’s  Budget  for  Los  Angeles” 
(Children’s  Planning  Council,  1999).  As  we  discuss  below,  CPC  members  would 
be  quite  active  in  developing  the  idea  of  a  unified  Plan  and,  as  important,  in 
developing  the  specific  LTFSS  Plan  that  emerged. 
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OVERVIEW  AND  DESCRIPTION  OF  THE  LTFSS  PLAN  PROCESS 

The  LTFSS  Plan  embodies  the  RBDM  Framework’s  approach  to  decision¬ 
making.  The  Plan  begins  with  the  Board's  vision  of  a  “holistic”  service  delivery 
system.  Rather  than  “figuring  out  how  to  comply  with  highly  prescriptive  federal 
and  state  regulations”,  the  Board  and  the  Plan  attempted  to  address  “the  most 
fundamental  question  ...  What  programs  and  services  will  best  help  CalWORKs 
and  other  low-income  families  achieve  long-term  self-sufficiency?”  The  formal 
planning  process  followed  many  of  the  steps  of  the  RBDM  Framework.  In 
addition,  the  Plan’s  Evaluation  Design  is  guided  by  the  RBDM  Framework’s 
methodology  and  requires  each  of  the  resulting  projects  to  undertake  evaluations 
using  such  RBDM  Framework  concepts  as  the  four  quadrant  performance 
measures,  baselines,  forecasts,  stories  behind  the  baseline,  and  the  comparison 
of  actual  outcomes  to  forecasts. 

In  this  section,  we  first  provide  a  brief  overview  of  the  environment  within 
which  the  LTFSS  Plan  was  developed  and,  in  particular,  the  budgetary  and 
programmatic  context  just  prior  to  its  development.  We  then  discuss  the  LTFSS 
Plan  development  process,  looking  at  the  planning,  implementation,  and 
evaluation  processes  separately. 

Budgetary  and  Programmatic  Context 

As  part  of  welfare  reform,  the  County’s  NDTF  conducted  a  broad-based 
effort  to  review  the  delivery  of  services  by  County  health  and  human  service 
agencies  to  families  enrolled  in  the  CalWORKs  program,  to  former  CalWORKs 
families,  and  to  other  low-income  families.  This  effort  resulted  in  the  LTFSS  Plan 
adopted  by  the  Board  on  November  16,  1999  (New  Directions  Task  Force, 

1999). 

The  LTFSS  Plan  itself  was  strongly  shaped  by  the  social  policy  context  that 
existed  at  the  end  of  the  1990s.  The  nation,  California,  and  the  County 
transitioned  from  a  recession  in  the  early  1990s  to  a  long  and  robust  economic 
expansion  through  2001 .  Employment  and  tax  revenues  were  rising,  and  the 
welfare  caseload  was  falling.  New  streams  of  funding  for  social  services— 
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Proposition  10,  the  tobacco  settlement,  new  mental  health  funding,  a  federal 
Medi-Cal  waiver— were  becoming  available. 

Nationally,  the  Personal  Responsibility  and  Work  Opportunity 
Reconciliation  Act  of  1996  (PRWORA)  and  in  California,  CalWORKs,  had 
radically  reformed  the  welfare  system.  A  combination  of  federal  funding  through 
block  grants  with  a  State  Maintenance  of  Effort  requirement,  rapid  caseload 
decline  (resulting  both  from  the  economy  and  from  welfare  reform  itself),  and 
slow  roll-out  of  expanded  County  programs  resulted  in  generous  funding  for 
State  and  County  welfare  operations.  (For  more  detail,  see  Klerman  et  al., 
2001 .)  Senior  DPSS  staff  noted  that  the  available  funding  was  in  excess  of  what 
the  County  projected  it  could  spend  on  the  conventional  welfare  and  programs 
on  WTW  services. 

In  addition  to  this  generous  funding  of  core  WTW  programs,  the 
CalWORKs  legislation  provided  a  second  source  of  funds.  Partially  to 
encourage  counties  to  move  recipients  into  the  labor  force  and  off  cash 
assistance,  the  CalWORKs  statute  provided  that  all  the  savings  resulting  from 
any  decline  in  aid  payments  were  to  be  returned  to  the  counties  in  the  form  of 
“Performance  Incentive  Payments”  (CalWORKs  Statute  S  10544.1). 

The  robust  economy  and  the  rapidly  dropping  caseload  led  to  the 
accumulation  of  such  Performance  Incentive  Funds  (PIF)  monies  well  in  excess 
of  any  initial  expectation.  By  early  1999,  the  County  had  “earned”  about  $400 
million  in  PIF  monies  (later  raised  to  about  $460  million).  In  fact,  after  the  LTFSS 
Plan  was  approved  and  partially  because  the  earned  PIF  money  was  so  large,  a 
sequence  of  actions  by  the  State  had  the  effect  of  deferring  and  then  essentially 
ending  the  accumulation  of  PIF  monies  and  delaying  the  actual  paying  of  the 
some  of  the  funds.  (See  County  Fiscal  Letters  No.  99/00-34  and  99/00-41 ,  and 
All  County  Letter  No.  00-57;  2000-01  budget  trailer  bill  for  social  services, 
Chapter  1 08,  Statutes  of  2000,  AB  2876  Aroner.) 

Description  of  the  Planning  Process 

These  funds  provided  the  opportunity  from  which  the  LTFSS  Plan 
emerged.  Figure  2.1  presents  the  timeline  for  the  development  of  the  LTFSS 
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Plan.  The  time  from  first  conceptualization  of  the  Plan  to  its  approval  is  less  than 
nine  months.  The  initial  spur  to  develop  the  Plan  appears  to  have  come  from 
Supervisor  Gloria  Molina  and  her  insistence  that  rather  than  have  the  available 
PIF  monies  doled  out  in  small  pieces  and  with  only  limited  oversight  from  the 
Board,  the  County  should  develop  a  unified  plan  for  the  use  of  these  funds 
aimed  at  “stabiliz[ing]  families  by  building  their  capacity  to  become  self- 
sustaining”  (Board  Minutes,  April  13, 1999).  That  vision  was  incorporated  into  a 
formal  Board  Resolution  of  April  13,  1999  “instruct[ing]  the  Director  of  Public 
Social  Services  to  immediately  delegate  to  the  [County’s]  New  Directions  Task 
Force  on  Welfare  Reform  (NDTF)  the  responsibility  of  creating  a  cohesive  ‘Long- 
Term  Family  Self-Sufficiency  Plan’”  and  requesting  a  report  back  within  60  days. 
This  NDTF  (originally  known  as  the  Welfare  Reform  Steering  Committee)  had 
originally  been  created  to  craft  the  County’s  reaction  to  welfare  reform  and 
included  representatives  of  many  County  departments. 


1999 


Apr  May  Jun  Ju!  Aug  Sep  Oct 


1  1  _ 1 _ 1 _ 1 _ 1 _ 

_ 

-4. 

n 

BoS  ii 
NDTF 
deveU 
LTFSS 

istructs 

to 

>P 

5  Plan 

Director  DPSS  and 
CAO  report  to  BoS 
on  planning  to  date 

- 1 - 1 - 

NE 

Ac 

LI 

i 

]JF  BoS 

Jopts  appi 

FSS  Plan  Lyp 

1 

roves 

SS  Plan 

Planning  Retreat  1 
to  identify 
outcomes 
and  measures 


Kickoff 
meeting 
for  5 

Workgroups 


Planning  Retreat  2 
to  review  projects 
proposed  by 
7  Workgroups 


Synthesis  of 
recommendations 
proposed  by 
7  Workgroups 


Figure  2.1— Timeline  for  Developing  the  LTFSS  Plan 

Following  the  initial  instruction  of  April  13,  NDTF  convened  a  small  group 
that  included  representatives  from  various  County  and  community  entities.  This 
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group’s  charge  was  to  draft  a  preliminary  list  of  outcome  indicators  that 
potentially  could  be  used  to  measure  long-term  family  self-sufficiency. 

On  May  21, 1999,  the  NDTF  convened  a  Planning  Retreat  to  develop  a  set 
of  measurable  indicators  of  “long-term  family  self-sufficiency.”  Participants  at  the 
Retreat  included  representatives  of  County  departments,  representatives  from 
the  CPC  and  from  each  SPA  Council,  and  selected  community  advocates  and 
researchers.  At  the  Retreat,  the  draft  list  of  45  potential  outcome  indicators  was 
presented.  Presentations  were  made  at  the  Retreat  on  the  available  funding  and 
using  Fiscal  Policy  Studies  Institute  (FPSI)  materials  on  Friedman’s  approach  to 
planning  and  evaluation  (i.e.,  the  RBDM  Framework).  Then,  following  the  steps 
of  the  RBDM  Framework,  Retreat  participants  selected  the  five  outcome  areas— 
good  health,  safety  and  survival,  economic  well-being,  social  and  emotional  well¬ 
being,  and  education  and  workforce  readiness— that  had  previously  been 
defined  by  the  CPC,  adopted  by  the  Board,  and  used  in  CPC’s  Children’s  Score 
Card.  Corresponding  to  these  five  outcome  areas,  they  identified  26  specific 
measurable  indicators.  That  list,  shown  in  Table  2.1,  served  as  the  basis  for  the 
final  set  of  indicators. 

On  June  14,  1999,  a  report  was  sent  to  the  Board  from  the  Director  of 
DPSS  and  the  Chief  Administrative  Officer  summarizing  the  findings  from  the 
May  Planning  Retreat  and  describing  the  creation  of  LTFSS  “Planning  for 
Results  Workgroups,”  (hereafter,  Workgroups)  one  for  each  of  the  five  outcome 
areas. 

The  kick-off  meeting  for  the  five  Workgroups  was  held  on  July  12,  1999. 
Their  charge  was  to  develop  services  that  would  positively  impact  the  identified 
measurable  indicators  of  long-term  family  self-sufficiency.  Chaired  by 
corresponding  County  agency  heads,  membership  on  the  Workgroups  included 
representatives  from  County  agencies,  service  providers,  other  public  agencies, 
advocates,  and  researchers.  During  the  process,  additional  Workgroups  for  teen 
services  and  cross-cutting  organizational  issues  were  established,  bringing  the 
number  of  Workgroups  to  seven. 


-  17- 


Table  2.1 

Initial  List  of  Outcome  Areas  and  Indicators  for  LTFSS  Plan 


Outcome  Area 

Indicators 

Good  Health 

Low  birth  weight  births  (-) 

Access  to  health  care  (+) 

Infant  mortality  (-) 

Births  to  teens  (-) 

Individuals  without  health  insurance  (-) 

Safety  and  Survival 

Domestic  violence  incidents  (-) 

Child  placement  in  out-of-home  care  (-) 

Juvenile  probation  violations  (-) 

Successful  minor/family  reunification  after  out-of¬ 
home  placement  (+) 

Youth  arrests  for  violent  crimes  (-) 

Economic  Well-Being 

Annual  income  under  Federal  Poverty  Level  (-) 

Adults  employed  by  quarter  (+) 

Percent  of  family  income  used  for  housing  (-) 

Access  to  transportation  (+) 

Adults  earning  a  living-wage  (+) 

Homeless  “episode”  within  prior  24  months  (-) 

Social  and  Emotional 
Well-Being 

Personal  behaviors  harmful  to  self  or  others 
(domestic  violence,  child  abuse/neglect, 
substance  abuse)  (-) 

Access  to  quality  child  care  (+) 

Participation  in  community  activities  (voting, 
volunteering,  mentoring,  church,  etc.)  (+) 
Parent-child  time  together  (+) 

Education  and 
Workforce  Readiness 

Teenage  high  school  graduation  (+) 

Adult  educational  attainment  of  high  school 
diploma,  GED,  or  eight  grade  reading  level  (+) 
Elementary  and  secondary  school  students  reading 
at  grade  level  (+) 

Mother’s  educational  attainment  at  child’s  birth  (+) 
High  school  graduation  among  mothers  who  gave 
birth  before  graduating  high  school  (+) 

Adult  participation  in  education  or  vocational 
training  (+) 

NOTE:  Parenthetical  sign  indicates  desired  direction  of  impact. 


The  Workgroups  were  given  broad  guidance  that  was  strongly  influenced 
by  the  RBDM  Framework: 
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•  Start  with  the  initial  list  of  indicators  and  develop  project 
proposals  tied  directly  to  one  or  more  of  the  indicators 

•  Develop  recommendations  in  each  of  the  outcome  areas  and 
proposals  to  address  programmatic  or  systemic  issues 

•  Revisit  the  list  of  measurable  indicators  and  suggest 
modifications  to  them,  only  after  having  derived  a  set  of 
recommendations  and  proposed  projects 

•  Develop  the  proposals,  when  possible,  through  a  consensus¬ 
building  process,  with  Workgroups  to  note  any  non-consensus- 
derived  recommendations 

•  Report  back  in  eight  weeks  with  recommendations. 

The  Workgroups  were  given  no  specific  guidelines  about  the  amount  of 
funding  available  for  each  of  the  specific  outcome  areas;  however,  the 
Workgroups  were  aware  of  the  overall  estimate  of  available  PIF  monies. 

On  September  16,  1999,  a  second  Planning  Retreat  was  held  in  which  the 
individual  Workgroups  reported  their  recommendations  and  project  proposals  to 
the  larger  group.  The  Workgroups  had  developed  67  proposals  that  were 
consolidated  and  refined  by  a  committee  of  20  County  and  non-County 
representatives  into  59  proposals,  of  which  55  had  a  “full  consensus.”  The  full 
package  of  59  proposals  was  forwarded  to  the  NDTF,  which  added  four 
additional  proposals  and  approved  the  resulting  package  on  October  5, 1999.  At 
this  point,  the  proposals  were  also  organized  into  eight  strategy  areas  for 
promoting  long-term  family  self-sufficiency.  Subsequently,  these  63  proposals 
were  combined  into  the  46  projects  that  make  up  the  LTFSS  Plan,  formally 
approved  by  the  Board  on  November  16, 1999. 

As  was  true  throughout  this  process,  the  final  Plan  clearly  reflects  the 
influences  of  the  RBDM  Framework  and  the  CPC,  beginning  with  explicitly  noted 
“common  themes”: 

•  “Where  possible,  services  to  families  should  support  the  family 
as  a  unit,  rather  than  focusing  on  individual  family  members  in 
isolation. 
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•  Just  as  individuals  live  in  families,  families  live  in  communities. 
Therefore,  strengthening  communities  is  an  important  element 
of  strengthening  families. 

•  Services  are  most  effective  when  integrated  at  a  community 
level. 

•  Focusing  on  positive  outcomes  for  families  is  key  to  delivering 
effective  services.” 

Also  consistent  with  the  RBDM  Framework  and  CPC’s  emphasis  on 
measurement  and  evaluation,  the  Plan  specifically  states  that  “[m]easurable 
outcomes  are  critical,  both  in  shaping  program  design/redesign  and  determining 
program  effectiveness.”  Thus,  each  proposal  was  required  to  have  “an  adequate 
evaluation  design  to  track  achievement  of  measurable  outcomes.” 

Many  other  “mandatory  elements”: 

•  does  not  duplicate  existing  services, 

•  does  not  supplant  other  funding, 

•  addresses  a  clearly  documented  need, 

and  some  of  the  “desirable  elements”: 

•  promotes  service  integration  and  does  not  increase  fragmentation, 

•  enhances  existing  systems, 

•  has  proven  to  be  effective, 

•  is  cost-effective, 

•  is  complementary  to  existing  programs, 

•  leverages  other  funding, 

•  has  a  positive  long-term  impact  for  participants), 

are  also  in  consonance  with  RBDM  Framework  and  CPC  themes. 

This  Plan  had  a  total  cost  of  approximately  $174  million  per  year,  i.e.,  $870 
million  over  the  five-year  planning  horizon  of  July  2000  to  June  2005.  About 
$201  million  of  those  funds  came  from  the  CalWORKs  Single  Allocation.  For  a 
variety  of  reasons,  County  expenditures  of  Single  Allocation  funds  were  well 
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below  the  budgeted  level,  allowing  for  full  funding  of  the  Single  Allocation 
components  of  the  LTFSS  Plan.  The  situation  with  respect  to  PIF  monies  was 
different. 

The  balance  of  the  funds,  about  $668  million,  or  about  $133.7  million  per 
year,  was  to  come  from  PIF  monies.  Available  PIF  monies,  however,  were  not 
sufficient  to  fully  fund  all  the  proposals  over  the  entire  five-year  period 
(approximately  July  2000  to  June  2005).  As  of  mid-1999,  the  best  estimate  of 
available  PIF  monies  was  only  slightly  more  than  $400  million.  Of  those  funds, 
about  $59  million  had  already  been  allocated  to  non-LTFSS  projects  (After- 
School  Enrichment  Program,  Child  Care  Grant  and  Loan  Program,  and  a  one¬ 
time  allocation  for  the  Child  Welfare  Services  (CWS)  Emergency  Assistance 
Program);  thus,  only  about  $345  million  (or  $69  million  per  year)  was  available 
for  the  LTFSS  Plan  over  the  five-year  planning  horizon. 

After  considering  various  options,  the  NDTF  (apparently  with  the  strong 
guidance  of  Lynn  Bayer  and  Phil  Ansell  of  DPSS)  recommended  that  the  14 
largest  projects  (those  requesting  at  least  $1  million  per  year)  be  funded  at  half 
the  initially  funded  level.  Table  2.2  lists  the  46  projects  and  their  initial  annual 
project  funding.  Projects  initially  requesting  under  $1  million  were  deemed 
indivisible  and  funded  in  full.  As  a  consequence,  the  six  projects  requesting 
between  $0.5  million  and  $1 .0  million  were  funded  at  a  level  higher  than  those 
requesting  $1  million:  Project  22,  Services  to  CalLEARN  and  Other  Teen 
Parents;  Project  24,  Public  Library  Services  for  Children  and  Youth;  Project  25, 
Operation  READ;  Project  28  Domestic  Violence  Prevention;  Project  40,  Strategic 
Planning  Data  Center;1  and  Project  41,  SPA  Council  Staff  and  Technical 
Assistance. 


1  Project  40  since  then  has  been  renamed  to  Data  Partnerships. 
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Table  2.2 

46  Projects  by  Strategy  and  Initial  Annual  Funding 


II.  CalWORKs  Welfare-to-Work  Strategy  _ I  $4.000,000 


12.  Employer-Linked  Education/Training  I  $2,500,000 


3.  Transitional  Subsidized  Employment/Paid  Work  Experience  _ I  $0 


4.  County  Apprenticeship  Program 

|5.  Business  Micro-Loan  and  Incubator  Program  for  CalWORKs  Participants  I  $1 .000,000 


7.  Strategic  Information  and  Technical  Assistance  to  Support  Targeted  Job 
Creation  Activities 

$325,000 

8.  Community  Economic  Development  Initiatives 

$500,000 

IHStliWACCESSTO  HEALTHCARE 


10.  Community  Outreach  to  Increase  Access  to  Health  Care  _ 


1 1 .  Hotline  to  Resolve  Health  Care  Access  Issues 


12.  Health  Care  Transportation  _ 


13.  “Health  First”  _  _ 


SUPPORTING  STABLE  HOUSIN 


14.  Transitional  Support  for  Homeless  CalWORKs  Families 


15.  Emergency  Assistance  to  Prevent  Eviction 


16.  Housing  Counseling/Trainin 


Wl^WiMBILPfNG  TEENS  BECOME  SELF-SUFFICIENT  ADULTS 


17.  Community-Based  Teen  Services  Program 


18.  Services  to  Teens  With  Special  Needs 


19.  Services  for  Emancipated  Foster  Youth  Who  are  Parents  _ 


20.  Teen  Passport  to  Success  Providers _ 

21 .  Staff  Development  for  Teen  Services  Providers _ 

22.  Services  to  CalLEARN  and  Other  Teen  Parents  _ 


23.  Youth  Jobs 


. .  PROMOTING  YOUTH  LITERACY 


24.  Public  Library  Services  for  Children  and  Youth 


25.  Operation  READ 


26.  Safe  Places 


27.  DART/STOP  For  CalWORKs  Families 


28.  Domestic  Violence  Prevention 


29.  School-Based  Probation  Supervision  _  _ 


30.  Support  Group  for  the  Families  of  Children  Aged  1 1  -1 8  on  Probation 


BUILDING  STRONG  FAMILIES  i  •  f!  ?!  3H/;-  £ 


31 .  Strategic  Support  for  Child  Care 


$5,000,000 


_ $0 


$ 


_ $0 


$8.800.000 


$3.750.000 


$500,000 


lawmia 

iMAMWihl 


$550,000 


$743,000 


$500,000 

$2.400.000 


$6,750,000 


$679,000 


$790,000 


$2.400,000 


$ 


$650,000 


$2,100,000 


$150.000 


$5,000,000 


33.  Family  Preservation 

$8,500,000 

34.  Home  Visitation  Program 

$5,250,000 

35.  Peer  Self-Help  Support  Groups 

$275,000 

36.  Support  and  Therapeutic  Options  Program  (STOP 


INTEGRATING  THE  HUMAN  SERVICES 


37.  School  Attendance  Areas  _ 


[38.  Multi-Disciplinary  Family  Inventory  and  Case  Planning  Teams 


39.  County  Family  Resource  Centers 


40.  Strategic  Planning  Data  Center  _ 


41.  Service  Planning  Area  Council  Staff  and  Technical  Assistance 


$100,000 


$7.000,000 


$575,000 


$637,000 
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Strategy/Project 

Funding 

42.  CalWORKs  System  Review 

$500,000 

43.  New  Directions  Lonq-Term  Family  Self-Sufficiency  Committee 

$0 

44.  CalWORKs  Case  Manaqement 

$0 

45.  TranStar  Enhancement 

$215,000 

46.  Lonq-Term  Family  Self-Sufficiency  Evaluation 

$2,000,000 

TOTAL 

$132,389,000 

NOTE:  Table  reflects  original  project  names. 


Description  of  the  Implementation  Process 

Understanding  how  the  Plan  was  implemented  in  the  County  is  essential 
for  understanding  how  useful  the  RBDM  Framework  is  as  something  other  than 
simply  a  guide  to  planning;  in  the  final  analysis,  the  County  adopted  the  RBDM 
Framework’s  vision  to  achieve  its  goal  of  long-term  family  self-sufficiency,  in  part 
by  changing  the  way  services  are  actually  delivered.  Here,  we  discuss  the 
timeline  over  which  the  Plan  was  put  into  practice  and  the  procedures  that  were 
put  into  place  to  do  so  and  about  the  context  in  which  this  occurred.  We  then 
turn  to  the  status  of  these  efforts  as  of  October  2001 . 

Key  Points  in  the  Implementation  Timeline.  Figure  2.2  provides  a 
timeline  of  key  dates  within  the  LTFSS  Plan  implementation  process.  When  the 
Board  approved  the  Plan  in  November  1999,  they  directed  the  NDTF  to  come 
back  to  the  Board  within  45  days  with  a  timeline  for  implementation.  The 
timeline  and  a  matrix  with  the  different  implementation  steps  were  developed 
jointly  by  the  LTFSS  planning  team  and  the  lead  agencies  and  were  submitted  to 
the  Board  in  December  1999.  At  that  time,  DPSS  also  initiated  project 
coordinators’  meetings  that  would  bring  together  representatives  from  each  of 
the  lead  agencies  on  a  regular  basis  to  meet  and  discuss  progress  on  the  LTFSS 
projects  and  any  implementation  issues  that  might  arise. 


BoS 

approves 
LTFSS  Plan 


Timeline  for 
implementation 
of  LTFSS 
Plan  developed 

DPSS  initiates 
project 
coordinators 
meetings 


Spring  00 

DPSS’s  oversight 
is  broadened  to 
include  coordination 
of  implementation 
of  projects  and 
fiscal  oversight 

DPSS  undergoes 
major 

reorganization 


DPSS  held 

initial 

Technical 

Advisory 

Workshop 


Target  date  for  projects 
to  start  spending  funds 


LTFSS  Division 
created; 

Monthly  projects 
coordinators* 
meetings 
continue 


Sprinq/Earlv 
Summer  01 


DPSS  held 
Technical 
Advisory 
Workshop 


LTFSS  Division 
hires  staff* 

LTFSS  Division 
initiates  one- 
on-one  meetings 
with  individual 
projects 


•LTFSS  Division  staff  were  already  in  place  when  staff  positions  were  allocated. 


Figure  2.2 — Implementation  Process  Timeline 

The  LTFSS  Plan  sent  to  the  Board  was  seen  as  a  collaborative  effort 
between  DPSS  and  the  other  lead  agencies,  with  the  initial  understanding  being 
that  DPSS  would  be  responsible  for  moving  dollars  to  the  other  lead  agencies 
and  for  implementing  the  specific  projects  it  had  been  designated  to  take  the 
lead  on.  Since  both  Single  Allocation  and  PIF  Funds  flowed  through  DPSS, 
some  coordinating  role  for  DPSS  was  inevitable.  Thus,  in  the  spring  of  2000, 
DPSS’s  role  expanded  to  include  programmatic  oversight  for  the  LTFSS  Plan’s 
implementation.  In  addition,  DPSS  established  a  system  of  project  coordinators 
that  would  serve  as  liaisons  to  the  lead  County  agencies.  DPSS  also  allocated 
staff  resources  internally  to  implement  the  specific  projects  for  which  it  served  as 
the  lead  agency.  All  of  this  occurred  simultaneously  with  a  major  reorganization 
in  DPSS. 

In  January  2001 ,  a  new  LTFSS  Plan  Division  was  created  within  DPSS  to 
consolidate  most  all  LTFSS-related  activities.  These  activities  included  both 
implementing  the  projects  for  which  DPSS  had  been  assigned  the  leading  role 
and  also  the  activities  required  by  DPSS’  new  oversight  role  for  LTFSS  Projects 
led  by  other  agencies.  With  the  establishment  of  the  LTFSS  Division,  DPSS 
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leadership  of  LTFSS  activities  shifted  from  the  Chief  of  the  Inter-Govemmental 
and  Inter-Agency  Relations  Division  to  the  Chief  of  the  LTFSS  Division.  The  shift 
resulted  in  some  loss  of  continuity  with  the  planning  process. 

In  September  2000,  DPSS  convened  a  Technical  Advisory  Workshop  to 
provide  lead  agencies  with  an  overview  of  the  approval  processes  for  project 
implementation,  contracts,  and  budgets.  Then,  in  April  2001,  DPSS  held  another 
Technical  Advisory  Workshop,  at  which  projects  were  provided  with  an  overview 
of  the  approval  process  for  the  Board  letter  and  for  memoranda  of  understanding 
(MOUs,  interagency  agreements  where  another  County  department  will  provide 
services  for  a  project),  or  contracts,  as  well  as  step-by-step  guidelines  on  how  to 
develop  the  various  required  implementation  documents  (i.e.,  Board  letter; 
implementation  plan;  line-item  budget;  MOUs;  and  contracts)  consistent  with 
DPSS  practice  and  the  required  DPSS  review.  In  the  summer  of  2001 ,  the  new 
LTFSS  Division  instituted  one-on-one  meetings  with  the  individual  projects  to 
help  them  develop  individual  elements  of  their  project  implementation  plans  and 
supporting  documents. 

Key  Elements  in  the  Implementation  Process.  The  LTFSS  Plan,  under 
the  influence  of  the  RBDM  Framework,  implied  that  a  management  structure  and 
corresponding  procedures  needed  to  be  established  to  manage  the  cross¬ 
department  efforts  before  the  projects  could  begin  to  deliver  services  to  County 
residents.  In  Chapter  4,  we  discuss  the  evolution  of  these  procedures  and  their 
effect  on  implementation  and  in  terms  of  what  they  imply  about  the  usefulness  of 
the  RBDM  Framework.  Here,  we  note  the  management  structure  and 
procedures  as  useful  background  material  both  for  that  discussion  and  for  the 
discussion  of  the  current  status  of  implementation  below. 

The  newly  created  LTFSS  Plan  Division  within  DPSS  plays  a  central  role  in 
administering  the  Plan’s  projects.  The  Division  is  responsible  for  DPSS’s  fiscal 
oversight  role.  It  is  also  responsible  for  coordinating  all  the  LTFSS  projects 
(internal  and  external  to  DPSS)  and  for  overseeing  the  review  and  clearance 
process  for  projects’  implementation  plans  and  MOUs  or  contracts.  The  review 
and  clearance  process  is  a  preliminary  step  prior  to  formal  Board  approval  of  a 
project.  Board  approval  of  a  project’s  implementation  plan  is  required  for  all 
projects  at  least  partially  funded  by  PIF  dollars  (i.e.,  for  the  majority  of  the  LTFSS 
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projects).  Board  approval  of  a  project’s  implementation  plan  is  also  required  for 
any  project  that  uses  an  outside  contractor  (for  greater  than  $25,000).2 

Once  the  Board  approves  a  project’s  implementation  plan,  a  project  can 
proceed  with  developing  its  MOUs  or  contracts.  Once  written,  contracts/MOUs 
may  require  separate  CAO,  County  Counsel,  and/or  Board  Approval.  LTFSS 
projects  that  involve  only  County  departments  (i.e.,  no  contractors)  and  are 
funded  solely  by  Single  Allocation  dollars  do  not  require  Board  approval.  For 
these  projects,  the  review  process  is  less  involved,  requiring  only  a  memo  to 
inform  the  Board  about  the  project’s  implementation  plan. 

The  documents  required  for  the  implementation  of  a  project  and  for  the 
approval  process  include  the  following:  (1)  Board  Letter  (jointly  signed  by  the 
lead  agency  and  DPSS);  (2)  an  implementation  Plan  that  describes  the  project, 
MOU  and  contractual  arrangements,  funding  and  estimated  costs,  how  the 
project  will  be  implemented,  target  population,  type  of  services  to  be  provided, 
and  scope  of  work;  (3)  a  line-item  budget;  and  (4)  MOU(s)  and/or  contract(s).  In 
addition,  all  projects  are  required  to  fill  out  an  immigrant  planning  guide  to  ensure 
that  the  special  needs  of  immigrants,  refugees,  and  limited  English  proficient 
(LEP)  participants  are  fully  addressed.  To  be  approved,  each  of  the  documents 
has  to  go  through  a  lengthy  and  detailed  review  and  clearance  process  by  all 
lead  agencies  involved  with  a  project’s  administration,  as  well  as  by  the  DPSS, 
the  CAO,  and  County  Counsel. 

Description  of  the  Evaluation  Design  Process 

The  Board  ultimately  wanted  to  know  who  the  LTFSS  Plan  projects  help, 
how  the  funding  was  used,  and  what  impact  the  projects  had  on  children  and 
families.  The  idea  of  results-based  accountability  underlying  the  RBDM 
Framework  and  thus  the  LTFSS  Plan  meant  that  each  project  should  be  able  to 
show  how  it  affected  specific  family  and  child  outcomes.  In  approving  the  Plan, 
an  important  Board  consideration  was  that  a  strong  and  independent  evaluation 
design  be  put  into  place. 


2LTFSS  Technical  Advisory  Workshop  Manual,  April  23,  2001 . 
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Figure  2.3 — Evaluation  Design  Process  Timeline 

As  Figure  2.3  suggests,  the  development  of  the  LTFSS  Plan’s  Evaluation 
Design  and  the  individual  project  evaluations  did  not  get  under  way  until  the 
creation  of  the  CAO/SIB  in  May  2000.  One  of  its  first  tasks  was  to  develop  the 
LTFSS  Plan’s  Evaluation  Design.  On  June  15,  2000,  the  CAO/SIB  convened  an 
Evaluation  Design  Development  Kick-off  Meeting,  bringing  together 
representatives  from  each  of  the  lead  County  agencies  to  begin  working  together 
on  developing  the  LTFSS  Plan  Evaluation  Design.  The  Evaluation  Design  was 
to  be  based  on  the  RBDM  Framework  model.  At  that  meeting,  Friedman 
provided  participants  with  an  overview  of  his  results-based  decision-making 
model  and  made  suggestions  about  how  the  Workgroup  might  approach  their 
task,  including  the  following: 

•  In  defining  “the  population,”  the  perspective  they  adopt  should 
first  be  the  County  as  a  whole  and  second  the  poor  families 
within  the  County 

•  The  Workgroup  should  narrow  down  the  list  of  indicators  and 
consider  choosing  3-5  of  the  most  important  26  outcome 
indicators  to  focus  on  initially 
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•  The  Workgroup  should  not  limit  the  evaluation  process  to 
County  departments;  instead,  it  should  also  involve  community 
partners,  such  as  community  agencies  and  welfare  rights 
advocates 

•  The  Workgroup  should  approach  evaluation  from  the  top  down 
and  bottom  up  simultaneously.  That  is,  Friedman  encouraged 
them  to  approach  evaluation  from  the  view  of  the  Board  and 
from  the  perspective  of  the  customers/participants.3 

Further,  the  LTFSS  Plan’s  Evaluation  Design  should  be  based  on  FPSI’s 
RBDM  Framework.  The  Evaluation  Design  Workgroup  members  also  received  a 
plan  summary  outline  listing  the  elements  that  should  be  addressed  in  the  draft 
Evaluation  Design,  such  as  collecting  and  analyzing  data  on  performance 
measures,  producing  reports  on  project-level  outcomes,  and  identifying  needed 
resources  and  technical  assistance.4 

The  Evaluation  Design  Workgroup  met  a  total  of  five  times,  with  the  goal  of 
submitting  an  Evaluation  Design  to  the  Board  by  November  2000  for  approval. 
In  December  2000,  the  Board  approved  the  LTFSS  Plan’s  Evaluation  Design.  At 
the  request  of  DPSS  and  the  LTFSS  Departmental  Coordinators,  the  CAO/SIB 
held  an  all-day  training  session  for  more  than  100  County  departmental 
representatives  to  introduce  them  to  the  Evaluation  Design  and  the  overall 
RBDM  Framework  approach.  At  that  session,  a  detailed  Project  Evaluation 
Implementation  Guide  was  provided  to  projects.  The  CAO/SIB  also  provided 
hands-on  technical  training  at  that  meeting  on  how  projects  should  implement 
the  Evaluation  Design.5 


3LTFSS  Evaluation  Design  Development  Kick-off  Workshop,  Meeting 
Notes,  June  15,  2000. 

4lbid. 

5The  LTFSS  Evaluation  Design  calls  for  the  continuation  of  the  Evaluation 
Design  Workgroup,  which  is  charged  with  continuing  to  guide  the  design  and 
implementation  of  the  LTFSSP  evaluation.  The  Workgroup  also  is  charged  with 
oversight  of  the  Countywide  evaluation  deliverables:  (a)  data  development 
agenda;  (b)  headline  indicators;  and  (c)  geographic  units  of  analysis  (County  of 
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The  lead  County  agency  is  responsible  for  a  project’s  evaluations, 
regardless  of  whether  other  County  departments  or  agencies  may  be  involved  in 
the  project.  With  respect  to  evaluation,  the  projects  themselves  have  six 
deliverables:  (1)  a  logic  model;  (2)  a  list  of  headline  and  secondary  performance 
measures;  (3)  a  data  sources  document;  (4)  a  data  development  agenda;  (5)  a 
story  behind  the  baselines  that  includes  a  progress  graph;  and  (6)  semi-annual 
reports  thereafter.  The  first  five  deliverables  are  preliminary  ones,  which  are 
compiled  and  condensed  into  the  sixth  deliverable,  the  semi-annual  report,  due 
approximately  1 1  months  after  a  project  is  implemented  and  every  six  months 
thereafter. 

The  project  evaluations  contribute  to  the  specific  goals  of  the  Evaluation 
Design6  by  serving  as  a  means  by  which  CAO/SIB  can  assess  the  progress  of 
individual  projects  and  help  identify  projects’  support  needs.  In  the  Spring  of 
2001,  CAO/SIB  established  an  Evaluation  Design  Panel  comprised  of 
representatives  from  the  DPSS  Evaluation,  Research,  and  Quality  Assurance 
Division,  Department  of  Health  Services  (DHS),  CPC,  implementing  agencies 
(e.g.,  contractors),  and  the  CAO.  It  oversees  a  formal  review  of  project 
evaluation  deliverables. 

Not  all  the  members  of  the  Evaluation  Design  Panel  had  been  previously 
involved  in  developing  the  LTFSS  Plan  Evaluation  Design.  Thus,  the  first  step 
was  to  orient  the  Panel  members  to  the  Evaluation  Design.  In  the  early  summer 
of  2001,  the  Panel  began  reviewing  projects’  draft  evaluation  deliverables.  The 


Los  Angeles  Long-Term  Family  Self-Sufficiency  Plan  Evaluation  Design,  page  3, 
October  23,  2000). 

6The  LTFSS  Evaluation  Guide  states  that  the  specific  goals  of  the  LTFSSP 
evaluation  are: 

•  to  evaluate  the  success  of  the  LTFSSP  in  terms  of  assisting  CalWORKs 
and  low-income  families  to  achieve  and  maintain  self-sufficiency  in 
relation  to  the  identified  outcomes,  vision,  and  key  project  elements; 

•  to  monitor  the  effectiveness  of  the  various  projects  and  sen/ices 
included  in  the  Plan  to  both  track  progress  and  to  guide  future 
programmatic,  fiscal,  and  operational  decisions; 

•  to  complement  other  Los  Angeles  County  evaluation  efforts  related  to 
initiatives  that  help  children  and  families  achieve  and  maintain  self- 
sufficiency. 
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review  is  an  iterative  process  between  the  projects,  CAO/SIB,  and  the  Panel  until 
a  project’s  evaluation  deliverables  are  finalized  and  approved. 

STATUS  OF  LTFSS  PROJECTS’  IMPLEMENTATION  AND  EVALUATION 

In  this  section,  we  discuss  the  current  status  (as  of  the  end  of  October 
2001)  of  the  lead  agencies  in  their  efforts  to  implement  and  evaluate  their 
projects. 

Status  of  Implementation 

Table  2.3  summarizes  the  implementation  status  of  the  various  projects  as 
of  the  end  of  October  2001.  The  table  includes  information  on  the 
implementation  status  of  not  only  the  46  projects  but  also  of  their  individual 
subcomponents. 

The  table  is  based  on  the  steps  in  the  review  and  clearance  process,  but  it 
expands  a  little  beyond  that.  We  subdivide  the  approval  process  into  four 
process  checkpoints:  (1)  whether  the  project  has  received  Board  memo/letter 
approval;  (2)  whether  the  implementation  plan  has  been  approved;  (3)  whether 
there  is  MOU  approval;  and  (4)  whether  there  is  approval  for  contracts/ 
subcontracts.  As  noted  above,  not  all  projects  require  approval  in  all  these 
areas. 

Assuming  projects  have  moved  beyond  the  approval  process,  the  next  step 
is  to  start  providing  services.  Of  course,  before  services  can  be  provided, 
projects  need  to  put  their  infrastructure  into  place  (e.g.,  hiring  staff,  hiring 
contractors,  putting  into  place  interagency  agreements,  finding  space  for  the 
program,  training  staff,  and  recruiting  program  participants).  This  distinction  is 
not  highlighted  in  the  table,  although  it  has  led  to  significant  delays  in  the  actual 
start-up  of  service  delivery.  Finally,  the  table  considers  whether  the  projects  are 
actually  spending  funds. 
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Table  2.3 

Implementation  Status  of  LTFSS  Plan  Projects 


Approval  Process 


Project  Name 

CtfWDWKs  WtW  Strategy 
Career  Plan  and  Prep  Seminar 
Enhanced  Appraisal 
Targeted  Initial  Job  Search 
Part-Time  Work  w/ 
Educ/Training 

Voluntary  Enhanced  Motivation 


Employer-Linked  Educ/Trainin 


County  Apprenticeship 
Proaram 


Business  Micro-Loan  and 
Incubator 

Business  Micro-Loan  Program 
Incubator  Without  Walls 


Housing  Relocation  Program 


Strategic  Info— Supp  Job 
Creation  _ 


Community  Economic 
Development  _ 


Mini-Career  Centers _ 


Community  Outreach — Health 
Care 

1931  (b)  Medi-Cal  Outreach 
Prenatal  Outreach 
CalWORKs  Family  Assistance 
Media  Outreach 
Improving  Inter-dept  Capacity 


ineto  Resolve  Health  Care 


Health  Care  Transportation 


Transitional  Support 
Homeless 

Relocation  Grant — Homeless 
Families 

Tenant-based  Trans.  Rental 
Asst. 

Trans.  Subsidized  Employment 


Housing  Counseling/Trainm 


Community-Based  Teen 
Services  ■ 


Teens  with  Special  Needs 


mancipated  Foster  Youth- 
Parents  ’  ■;  '  _ _ 


Teen  Passport  to  Success 


Staff  Development  for  Teen 
Service  Providers 


Cal-Learn  and  Teen  Parents 


Board 

Memo/  Imp 

Letter  Plar 

App.  App 


Contr/  Svc  Have 

Sub-  Delivery  Spent 

contr  Start-up  Funds 
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No.  Project  Name 


Teen  Career  Enhancement 
Career  Counselors 


Youth  Jobs 


Public  Library  Svcs  for  Children 
Homework  Center 
Teen  Library  Card  Campaign 
Support  Services  for  After- 
School  Program 


Safe  Places 


DART/S 


Domestic  Violence  Prevention 
Risk  Assessment  Tool 
Research:  What  Stops  DV 
DVTeen  Curriculum 


Board 

Memo/ 

Letter 

App. 


Approval  Process 


Contr/ 

Sub- 

contr 


Svc 

Delivery 

Start-up 


Have 

Spent 

Funds 


Support — Families  of  Probation 
Child 


Federal  Family  Supp  Svcs 
Network 


LTFSS  Family  Preservation 


Home  Visitation  Program 
Nurse  Home  Visitation  Prog 
Home  Visitation  and  Case 
Management 


Peer  Self-Help  Support  Grgu 


Support  and  Therapeutic 
Options 


Multi-Disciplinary  Family 
Invento 


Data  Partnerships 


CalWORKs  Systems  Review 


CalWORKs  Case  Management 


TranStar  Enhancement 


L-T  Family  Self-Suff  Evaluation 


Deleted:  Funding  provided  by  different  source 


X 


N/A 

'M . -  ■  1 

X 

N/A 

N/A 

X 

urn »im 

N/A 

N/A 
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As  the  table  shows,  the  project  roll-out  is  proceeding  slowly.  As  of  October 
2001 7,  17  projects  and  4  subcomponents  have  been  implemented,  i.e.,  they 
have  reached  the  point  of  being  able  to  start  up  services  (the  Xs  in  the  second- 
to-last  column).  Five  of  the  17  projects  at  this  stage  are  projects  that  do  not 
provide  services  to  recipients:  12,  37,  41,  43,  and  46.  Thus,  considering  only 
those  projects  providing  direct  services,  we  are  left  with  only  12  projects  and  4 
subcomponents  that  have  begun  delivering  services  to  the  residents  of  the 
County. 

As  noted,  the  final  stage  of  implementation  for  most  (but  not  all)  projects 
involves  spending  funds.  For  example,  seven  of  the  20  projects  that  have 
reached  the  point  of  start  up  of  sen/ices  do  not  have  any  initial  monetary 
allocation:  3,  4,  11,  12,  13,  43  and  44  (Table  2.2,  above).  Another  11  projects 
have  reached  the  point  of  being  able  to  start  up  services  and  start  spending  their 
monetary  allocations,  and  are  spending  funds.  Of  the  1 1  projects,  9  provide 
direct  services  to  recipients. 

When  and  how  much  those  projects  are  spending  is  summarized  in  Table 
2.4.  It  tabulates  the  most  recent  available  data  covering  the  period  through  June 
2001  (the  end  of  Fiscal  Year,  FY,  2000-2001).  For  budgeting  purposes,  the 
original  project  budgets  assumed  (except  for  some  small  one-time  costs) 
constant-level  expenditures  for  the  period  July  2000  to  June  2005.  Actual 
expenditures  have  followed  a  very  different  pattern. 

Two  projects — Project  29,  School-Based  Probation  Supervision;  and 
Project  34(a)  Nurse  Home  Visitation  Program — started  spending  funds  in  late  FY 
1999-2000  (i.e.,  before  July  2000).  These  two  projects  appear  to  have  spent  at 
approximately  their  budgeted  levels.  Another  project  (Project  23,  Youth  Jobs) 
began  in  FY  2000-2001  and  is  spending  at  approximately  its  budgeted  level.  It 
alone  accounts  for  nearly  half  the  expenditures  to  date  ($10.4  million  of  the 
combined  FY  1999-2000  and  FY  2000-2001  total  of  $22.1  million). 


7ln  assessing  where  the  projects  are  in  the  process,  we  relied  on 
interviews,  LTFSS  Project  Status  Update  charts,  and  other  sources  during  the 
process  assessment,  information-gathering  stage,  which  ended  in  October  2001 . 
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Table  2.4 

LTFSS  Plan  Expenditures  by  Month  ($) 


Proj# 


6/00 


Actuals 

PV  95*00 


10/00 


11/00 


12/00 


1/01 


2/01 


3/01 
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5/01 
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1 Actuate  ’ 
rvwi 
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25 

29 

34a 

38 
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505,588  i!:!m§3&8 
954,476  WMmmmtii 


1,500 


405 


254,945 


2,305 


644,116 

603,371 


2,981 


36,861 

324,566 

273,834 


50 


22,255 

183,983 

43,100 

30.271 


2,350 


21,534 

283,096 

203,440 

59,600 


VgARLV  TOTAL 


Interpreting  the  spending  levels  of  the  other  projects  is  complicated  by  the 
fact  that  the  billing  patterns  from  month  to  month  are  far  from  level.  While  we 
might  expect  a  slow  ramp-up  of  monthly  expenditures,  for  some  projects  the  first 
month  with  recorded  expenditures  is  much  higher  than  it  is  for  subsequent 
months  (e.g.,  Project  29,  where  the  opposite  pattern— slow  ramp-up— might 
have  been  expected),  suggesting  that  expenditures  were  held  and  then  billed  as 
a  bunch.8  Given  those  caveats,  three  other  projects  started  spending  toward  the 
end  of  FY  2000-2001  and  appear  to  have  expenditures  that  might  put  them  near 
the  budgeted  levels  on  an  annualized  basis — Project  9,  Mini-Career  Centers; 
Project  25,  Operation  Read;  and  Project  38,  Multi-disciplinary  Family  Inventory. 
Another  five  projects  have  some  expenditures,  but  the  limited  available  evidence 
suggests  that  those  expenditures  are  well  below  the  budgeted  level — Project  4, 
County  Apprenticeship  Program;  Project  6,  Housing  Relocation  Program;  Project 
17,  Community-Based  Teen  Services  Program;  Project  41,  SPA  Council  Staff 
and  Technical  Assistance;  and  Project  46,  LTFSS  Plan  Evaluation.  A  few 
projects  currently  exist  under  different  funding  sources  and  will  be  picked  up  by 
the  LTFSS  Plan  at  a  later  date  (e.g.,  Project  10).  Finally,  as  noted  earlier,  the 
remaining  projects  with  monetary  allocations  have  no  expenditures  at  all. 


8Projects  14  and  15  had  their  single  allocation  funding  eliminated,  with  PIF 
monies  subsequently  allocated  for  these  projects  in  a  February  2001  Board 
motion.  In  the  case  of  Project  29,  the  program  was  implemented  in  March  2000, 
but  the  MOU  was  not  finalized  by  DPSS  until  January  2001 .  Once  that  process 
was  finalized,  the  lead  County  agency  billed  for  the  full  period. 
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Status  of  Evaluation 

As  was  true  for  implementation  and  expenditures,  relatively  few  project 
evaluation  formative  deliverables  have  been  submitted  and  even  fewer  have 
been  reviewed  after  initial  Evaluation  Design  Panel  comments.9  As  of  October 
2001,  21  projects  had  evaluation  deliverables  due.  As  shown  in  Table  2.5,  16 
projects’  draft  evaluation  formative  deliverables  have  been  submitted  for  review 
and  of  those  16,  six  have  been  approved  by  the  Evaluation  Design  Panel: 
Project  4,  County  Apprenticeship  Program;  Project  22a,  Teen  Career 
Enhancement;  Project  23,  Youth  Jobs;  Project  25,  Operation  READ;  Project 
34a:  Nurse  Home  Visitation  Program;  and  Project  38:  Multi-Disciplinary  Family 
Inventory.  In  addition,  three  projects  have  received  partial  approval  of  their  draft 
evaluation  formative  deliverables.  That  is,  two  out  of  the  three  formative 
deliverables  of  these  projects’  evaluations  (logic  model,  performance  measures, 
and  data  sources  document)  have  been  approved.  These  three  projects  are: 
Project  2,  Employer-Linked  Education/Training;  Project  29,  School-based 
Probation  Supervision;  and  Project  41,  SPA  Council  Staff  and  Technical 
Assistance. 

As  noted,  16  projects’  evaluation  formative  deliverables  have  been 
submitted  for  review.  Other  projects  have  also  made  progress  in  developing 
their  evaluation  formative  deliverables  (e.g.,  selecting  performance  measures, 
developing  a  logic  model,  identifying  potential  data  sources  and  information 
systems  requirements),  but  they  have  not  yet  focused  on  the  development  of 
their  overall  project  evaluation,  nor  have  they  yet  submitted  any  individual  project 
evaluation  formative  deliverables  for  approval. 

In  assessing  the  current  status  of  projects’  evaluation  formative 
deliverables,  it  is  important  to  keep  in  mind  the  broader  context  of  the  overall 
evaluation  process.  As  shown  earlier  in  Figure  2.3,  although  the  Board  approved 
the  LTFSS  Plan  in  November  1999,  the  Evaluation  Design  itself  was  not  put  into 


9By  formative  deliverables,  we  mean  that  projects  have  selected 
performance  measures,  developed  a  logic  model,  and  identified  potential  data 
sources  and  information  systems  requirements. 
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place  until  December  2000.  A  review  process  then  had  to  be  put  into  place  and 
training  sessions  conducted  on  the  Evaluation  Design. 

An  additional  important  reason  for  the  delay  in  the  development  of  projects’ 
evaluation  formative  deliverables  is  that  the  lead  County  agencies  have  tended 
to  focus  initially  on  developing  their  implementation  plans,  getting  those  plans 
through  the  Board  approval  process,  and  creating  the  infrastructure  to  deliver 
services.  Projects  believed  that  it  was  necessary  to  get  their  implementation 
plans  into  place  first  before  they  could  start  developing  their  evaluations, 
particularly  those  projects  involving  contractors.  As  discussed  earlier, 
development  of  project  implementation  plans  has  been  a  slower  and  more 
complex  process  than  initially  anticipated,  given  initial  budgets  and 
implementation  timelines.  Given  the  sequencing — implementation  and  only  then 
development  of  the  evaluation  plan— the  longer  timeline  for  implementation  also 
pushed  back  the  development  of  evaluation  plans. 
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Table  2.5 

Status  of  Projects’  Evaluation  Formative  Deliverables 


Project  Name 


CalWQRKs  WtW  Strategy 
Career  Plan  and  Prep  Seminar 
Enhanced  Appraisal 
Targeted  Initial  Job  Search 
Part-Time  Wo*  w/  Educ/Training 
Voluntary  Enhanced  Motivation 


Employer-Linked  Educ/Trainin 


County  Apprenticeship  Program 


Business  Micro-Loan  and  Incubator 
Business  Micro-Loan  Program 
Incubator  Without  Walls 


Housina  Relocation  Program 


Strategic  Info— Supp  Job  Creation 


Community  Economic  Development 


ini-Career  Centers  -  ■ 


Community  Outreach — Health  Care 
1931  (b)  Medi-Cal  Outreach 
Prenatal  Outreach 
CatWORKs  Family  Assistance 
Media  Outreach 
Improving  Inter-dept  Capacity 


ne  to  Resolve  Health  Care 


Health  Care  Transportation 


iwtfi."  - _ ■ 


Transitional  Support — Homeless 
Relocation  Grant — Homeless  Families 
Tenant-based  Trans.  Rental  Asst. 
Trans.  Subsidized  Employment 


istance— Eviction 


Housing  Counselin 


Community-Based  Teen  Services 


Teens  with  Special  Needs 


Prior  to 
Submittal 


Submitted 
for  Review 


\mm&& 


Teen  Passport  to  Success 


Staff  Development  for  Teen  Service 
Providers 


Cal-Leam  and  Teen  Parents 
Teen  Career  Enhancement 
Career  Counselors 


Youth  Jobs  J  :  i  v . 


Public  Library  Svcs  for  Children 
Homework  Center 
Teen  Library  Card  Campaign 
Support  Services  for  After-School 
Program 


ration 


Safe  Places 


DART/STOP 


Domestic  Violence  Prevention 
Risk  Assessment  Tool 
Research:  What  Stops  DV 
DV  Teen  Curriculum 


ased  Probation  Supervision 


Support — Families  of  Probation  Child 
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No. 

Project  Name 

Prior  to 
Submittal 

Submitted 
for  Review 

Resubmitted 

Approved 

31 

93ia| 

31b 

St«rfegte3upport  for  £hUd  d§re,  , 

Jtifti  Care—Norvneedy  Caregiver 

Inc  Non-traditional  Child  Care 

a 

||  J t"[ 

32 

Federal  Family  Supp  Svcs  Network 

X 

mm 

iiTFSB  family  PreservaBorM  lliilMsf 

§1"  ir  I99 

wm  f  m  mm 

34 

34a 

34b 

x 

X 

38 

Self-Help  Support  Groups 

3885! 

36 

Support  and  Therapeutic  Options 

Deleted;  Funded  from  a  separate  source  ! 

Iff- 

■•V ■ 

. 

38 

Multi-Disciplinary  Family  inventory 

■m 

39 

39a 

t39b 

■■■ 

*f;  §|ij 

111  '■  11 

■B 

40 

Data  Partnerships 

X 

SPA  Council  Staff  and  Tech  Asst,  * 

8—1 

— — 

42 

CalWORKs  Systems  Review 

■miH 

■■■ 

mm 

s,:  mwm. 

.  1 

44 

CalWORKs  Case  Management 

X 

■■■ 

*3 . ;i 

MM^BHIiiarnent  - 

46 
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NOTE:  There  are  three  evaluation  formative  deliverables  for  the  project  evaluations  that  must  be 
approved:  logic  model,  performance  measures,  and  data  sources  document.  If  two  of  these  deliverables 
have  been  approved  and  the  third  deliverable  is  still  pending,  we  use  the  letter  “P”  to  designate  partial 
approval.  For  example,  Project  2  has  had  two  of  its  evaluation  formative  deliverables  approved,  with  the 
third  evaluation  formative  deliverable  having  been  resubmitted  to  the  Evaluation  Design  Panel  for 
consideration.  On  Project  29,  two  of  its  evaluation  formative  deliverables  have  been  approved  by  the 
Evaluation  Design  Panel,  with  the  third  evaluation  formative  deliverable  just  recently  submitted  for  review. 

DISCUSSION 

This  chapter  has  provided  an  overview  of  the  RBDM  Framework  and  the  LTFSS 
Plan  and  outlined  the  policies  and  procedures  that  evolved  over  time  to  turn  an 
idea  into  an  operational  County  effort  to  help  families  achieve  long-term  self- 
sufficiency.  We  discussed  how  the  planning  process  led  to  the  LTFSS  Plan  and 
how  the  next  steps  of  the  process— implementing  and  evaluating  projects — are 
still  under  way.  This  background  information  sets  the  context  for  our  analysis  of 
the  utility  of  the  RBDM  Framework  in  the  County  in  the  subsequent  chapters, 
because  an  assessment  of  how  useful  the  RBDM  Framework  was  must  start 
with  how  the  RBDM  Framework  was  actually  used.  Those  experiences, 
described  in  the  next  chapter  and  elaborated  upon  in  later  chapters,  are  an 
indispensable  source  of  insight  and  knowledge  on  which  to  build  as  the  County 
considers  how  it  might  adapt  and  refine  the  RBDM  Framework  for  future 
applications. 
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3.  FINDINGS  ABOUT  THE  UTILITY  OF  THE  RBDM  FRAMEWORK  IN  THE 

PLANNING  PROCESS 


INTRODUCTION 

The  RBDM  Framework  provides  clear  guidance  about  how  to  conduct 
strategic  planning  and  the  RBDM  Framework  was  used  in  developing  the  LTFSS 
Plan.  The  previous  chapter  described  the  development  of  the  Plan  and  the 
status  of  implementation  and  evaluation.  In  this  chapter,  we  report  the 
impressions  of  our  key  informants  about  the  use  and  utility  of  the  RBDM 
Framework  in  the  planning  phase  (i.e.,  the  development  of  the  Plan).  We 
consider  how  the  RBDM  Framework  was  used  in  practice,  what  issues  arose, 
how  they  were  addressed,  and  what  changes — based  on  this  experience — 
participants  thought  future  applications  of  the  RBDM  Framework  in  Los  Angeles 
County  should  consider. 

THE  UTILITY  OF  THE  RBDM  FRAMEWORK  IN  THE  PLANNING  PROCESS 
The  RBDM  Framework  Succeeds  in  Focusing  Attention  on  Ultimate  Results 

As  discussed  in  Chapter  2,  the  RBDM  Framework  urges  planners  to  begin 
by  identifying  the  result(s)  or  outcome(s)  they  want  to  improve  and  to  identify  a 
list  of  outcome  indicators  that  quantify  the  achievement  of  the  outcomes.  It  then 
urges  planners  to  choose  projects  that  they  believe  will  improve  those  outcomes. 
Further,  the  RBDM  Framework  urges  planners  to  involve  all  the  relevant 
stakeholders  in  a  collaborative  process  in  deciding  which  result(s)  and 
outcome(s)  the  Plan  should  achieve  and  to  select  indicators  to  measure  progress 
toward  the  outcomes. 

In  practice,  the  LTFSS  planners  followed  the  RBDM  Framework  in  defining 
the  Plan’s  overall  objective  of  helping  families  to  achieve  long-term  self- 
sufficiency.  Following  Friedman’s  guidance,  in  the  spring  of  1 999,  a  small  task 
force  convened  by  DPSS  drafted  a  preliminary  list  of  45  outcome  indicators  that 
might  be  used  to  measure  long-term  family  self-sufficiency.  This  draft  set  of 
indicators,  along  with  the  RBDM  Framework,  were  presented  at  a  May  1999 
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Planning  Retreat  that  brought  together  a  diverse  group  of  individuals  from  inside 
and  outside  County  government.  They  included  representatives  from  County 
departments,  major  agencies  outside  of  the  County  (e.g.,  the  Los  Angeles 
Unified  School  District,  LAUSD),  the  City  of  Los  Angeles,  the  CPC,  SPA 
Councils,  and  selected  community  advocates  and  researchers.  Retreat 
participants  were  asked  to  help  develop  a  definition  of  “long-term  family  self- 
sufficiency”  and  to  modify  the  draft  set  of  measurable  indicators  based  on  the 
RBDM  Framework. 

Retreat  participants  reached  a  consensus  on  the  definition  of  long-term 
family  self-sufficiency  and  on  reducing  the  draft  list  of  45  outcome  indicators  to  a 
set  of  26  indicators  to  measure  progress  toward  the  goal  of  achieving  self- 
sufficiency  for  children  and  families.  The  26  indicators  were  then  grouped  into 
five  outcome  areas — good  health,  safety  and  survival,  economic  well-being, 
social  and  emotional  well-being,  and  education  and  workforce  readiness — that 
had  previously  been  defined  by  the  CPC,  adopted  by  the  Board,  and  used  in 
CPC’s  Children’s  Score  Card.  Subsequent  discussions  about  what  programs 
would  help  the  County  achieve  its  goal  of  self-sufficiency  for  low-income  families 
was  then  guided  by  this  list  of  indicators. 

In  short,  many  of  the  steps  in  developing  the  LTFSS  Plan  followed  the 
procedures  of  the  RBDM  Framework  and  the  RBDM  Framework  proved  to  be  a 
useful  planning  tool.  The  planning  process  brought  a  wider  array  of  perspectives 
to  the  table  than  has  been  the  case  in  other  planning  efforts,  this  new 
collaborative  approach  helped  the  County  think  broadly  about  what  it  wanted  to 
achieve.  Only  after  setting  overarching  goals  would  the  County  turn  to  the 
methods  for  achieving  those  goals. 

While  using  the  RBDM  Framework  was  reported  by  interviewees  to  have 
improved  planning,  we  also  heard  that  changes  in  the  way  the  Framework  is 
applied  could  make  it  even  more  useful.  The  following  sections  discuss  those 
areas  where  changes  in  the  application  of  the  RBDM  Framework  would  make  it 
even  more  useful  in  planning. 
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Applying  the  RBDM  Framework  Would  Have  Been  Facilitated  by  a  Longer 
Planning  Process 

The  RBDM  Framework  does  not  specify  a  time  frame  over  which  planning 
should  be  accomplished,  only  that  planning  is  an  iterative  process,  in  which 
results  feed  back  into  additional  planning  and  efforts  to  refine  the  overall  Plan 
and  its  component  projects.  The  County  used  its  previous  experience  planning 
welfare  reform  as  a  guide,  allocating  six  months  for  overall  planning,  including 
eight  weeks  for  developing  proposals.10  The  LTFSS  planning  leaders  noted  that 
this  choice  of  a  short,  intensive  planning  process  was  based  on  experience  with 
planning  for  welfare  reform.  There,  a  short,  intensive  planning  process  led  to 
quality  products  while  keeping  the  engagement  of  the  key  players.  Given  this 
experience,  these  leaders  used  a  similar  planning  process  for  the  LTFSS  Plan. 
They  felt  that  six  months  was  the  longest  period  of  time  they  could  ask  for  to 
accomplish  this  task. 

Following  the  May  Planning  Retreat,  five  Workgroups  were  created 
corresponding  to  the  five  outcome  areas.11  The  Workgroups  were  given  eight 
weeks  to  develop  project  proposals  and  recommendations  in  the  five  outcome 
areas  that  are  tied  to  the  list  of  26  Countywide  indicators.12  This  focus  on 
ultimate  outcomes  is  in  keeping  with  the  RBDM  Framework’s  intent  and,  for  the 
most  part,  appears  to  have  followed  the  planning  guidance  outlined  by  the 
RBDM  Framework.  Nevertheless,  when  interviewed,  most  Workgroup 
participants  did  not  explicitly  know  about  the  RBDM  Framework  or  recalled  using 
it. 


10The  entire  planning  process  for  developing  the  LTFSS  Plan  spanned  a 
little  over  six  months,  starting  April  13  with  the  Board’s  instructions  to  the  NDTF 
to  begin  the  process  and  ending  October  5  with  the  NDTF’s  adoption  of  the 
approved  Plan. 

^Subsequently,  two  additional  Workgroups  were  created  to  address  the 
needs  of  teens  and  integration  of  services. 

12The  steps  of  this  process  are  described  in  more  detail  in  Chapter  2.  The 
Workgroups  had  no  specific  guidelines  about  the  amount  of  funding  available  for 
each  of  the  specific  outcome  areas;  however,  they  knew  the  overall  estimate  of 
available  PIF  monies. 
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Looking  back  on  their  experiences  with  the  LTFSS  Plan,  many 
interviewees  thought  that  in  future  applications  of  the  RBDM  Framework,  the 
planning  and  project  selection  process  would  benefit  from  more  time.  They  felt 
more  time  would  lead  to  a  number  of  improvements:  even  wider  community 
participation  in  planning,  more  thorough  discussion  of  all  possible  options  for 
spending  the  funds,  more  attention  paid  to  the  Plan’s  strategy  of  integrating  the 
human  services  delivery  objectives  and  how  new  projects  fit  into  existing  services 
in  the  County,  and  more  time  to  bring  research  evidence  to  bear  on  project 
efficacy  and  cost-effectiveness.  We  discuss,  in  turn,  the  context  for  each  of 
these  areas  for  improvement. 

The  RBDM  Framework  advocates  broad  community  involvement  in  the 
planning  process  and  calls  for  ensuring  that  all  the  key  stakeholders  are  at  the 
table.  While  there  does  appear  to  have  been  broad  involvement  of  the 
community  and  non-County  stakeholders  in  the  May  21,  1999,  meeting 
convened  by  the  NDTF  to  develop  a  set  of  measurable  indicators  of  “long-term 
family  self-sufficiency,”  the  Workgroup  planning  process  that  started  in  July  1999 
appears  to  have  been  less  successful  in  maintaining  a  high  level  of  community 
involvement  as  these  ideas  were  transformed  into  the  final  plan  between  July 
and  November  1999.  Further,  there  does  not  appear  to  have  been  strong 
overlap  between  the  May  Planning  Retreat  participants  and  those  involved  in  the 
Workgroup  planning  process  itself,  so  that  there  was  often  little  institutional 
memory  of  earlier  community  input.  The  resulting  divergence  between 
community  input  and  final  Workgroup  decisions  appears  to  have  frustrated  some 
of  the  community  participants  involved  in  the  initial  planning  process. 

The  degree  of  community  involvement  also  appears  to  have  varied  across 
the  Workgroups.  For  example,  about  a  third  of  Workgroup  planning  participants 
(including  representatives  of  community  groups,  academics,  and  County 
employees)  interviewed  felt  that  the  meetings  and  the  selection  of  proposed 
projects  was  dominated  by  individual  County  departments.  Further,  those 
participants  usually  felt  that  the  process  of  assigning  a  lead  agency  for  each 
project  was  also  dominated  by  the  County  departments.  In  addition,  they  felt 
that  DPSS  had  a  strong  influence  on  the  final  selection  of  projects.  Then  again, 
six  participants  felt  that  community  advocacy  groups  in  several  Workgroups  had 
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particularly  strong  voices  in  the  planning  process  and  were  successful  in  getting 
their  proposed  projects  adopted.  In  one  instance,  an  advocacy  group 
recognizing  the  importance  of  getting  lead  agency  buy-in  felt  that  it  had  worked 
closely  with  a  Workgroup  chair  to  garner  support  within  the  Workgroup  and  with 
DPSS  for  the  group’s  proposal. 

The  second  suggested  area  that  could  benefit  from  an  extended  planning 
period  concerned  funding  decisions.  Some  Workgroup  participants  suggested 
that  because  of  the  short  planning  phase,  alternative  views  were  not  fully 
addressed.  For  example,  several  interviewees  wanted  PIF  monies  to  be  used  to 
fix  problems  with  existing  programs  or  to  expand  existing  programs  rather  than 
develop  new  projects.  Another  view  expressed  by  interviewees  was  that  PIF 
monies  perhaps  ought  to  be  concentrated  in  a  few  areas,  for  example,  either  to 
fund  fewer  well-developed  programs  or  to  target  specific  programmatic  areas 
such  as  housing,  health,  or  child  care.  A  third  view  expressed  was  that  PIF 
monies  should  be  used  to  build  a  better  safety  net  that  would  benefit  the  largest 
number  of  families  possible  rather  than,  in  their  view,  focusing  on  specific 
subgroups.  A  fourth  view  was  that,  given  the  unavailability  of  data  for  many  of 
the  selected  outcome  indicators,  a  smaller  set  of  indicators  (e.g.,  10-15)  should 
have  been  selected.  The  point  is  not  that  one  view  was  right  and  another  wrong, 
but  that  reconciling  opposing  views  and  building  consensus,  as  the  RBDM 
Framework  guides,  takes  considerable  time,  time  that  was  not  available  given 
the  short  planning  period. 

The  third  suggested  area  for  improvement  concerned  the  interrelation  of 
the  projects.  Those  involved  in  the  Workgroup  planning  process  felt  that  the 
short  planning  time  frame  did  not  allow  for  full  consideration  of  how  the  Plan’s 
projects  fit  together.  The  RBDM  Framework  states  that  as  part  of  the  planning 
process,  it  is  important  to  “fit  the  pieces  together”  and  to  consider  how  proposed 
programs  or  approaches  fit  together  into  a  system  of  services  and  supports,  “not 
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just  a  loose  confederation  of  good  ideas.”13  The  results  of  the  planning  process 
ideally  should  be  reviewed  by  the  broader  community. 

Interviewees  commented  that  there  was  little  time  to  integrate  proposals  at 
the  end  of  the  Workgroup  planning  process.  The  Workgroups  had  been  set  up 
to  address  each  of  the  outcome  areas,  with  each  group  tasked  to  develop 
recommendations  and  proposals  to  address  both  programmatic  and  systemic 
problems  in  each  area.  However,  as  noted  by  five  Workgroup  participants  and 
planning  leaders,  the  short  time  frame  within  which  they  had  to  do  this  work  left 
little  time  for  cross-fertilization  of  ideas  between  Workgroups  nor  time  to  fully 
integrating  similar  concepts. 

Interviewees  from  the  Workgroup  planning  process  also  commented  that 
there  was  little  time  to  consider  how  to  integrate  LTFSS  projects  with  existing 
service  delivery  systems.  Some  observers  were  concerned  that  because  LTFSS 
projects  were  overlaid  on  top  of  existing  service  delivery  systems,  the  integration 
of  LTFSS  Plan  services  with  existing  programs  might  be  incomplete,  and  that  not 
having  existing  service  providers  more  involved  in  planning  would  lead  to 
integration  problems  during  implementation. 

Interviews  with  lead  agency  and  project  staff  suggest  that  some  such 
problems  did  arise  in  the  project  implementation  phase.  For  example,  staff  of 
several  lead  agencies  expressed  concerns  that  LTFSS  projects  would  be  making 
referrals  to  an  already  overburdened  service  provider  network.  Similarly,  as  four 
lead  agency  staff  commented,  although  referrals  can  be  made  to  other  programs 
within  the  County,  if  it  still  takes  a  participant  4-6  weeks  to  get  an  appointment 
with  a  treatment  provider  or  eligibility  worker,  then  the  system  breaks  down. 
Lead  agency  interviewees  noted  that  community  service  providers  felt  that  some 
of  the  LTFSS  projects  duplicated  existing  services.  Staff  from  several  other  lead 
agencies  remarked  on  LTFSS  projects  that  in  their  view  could  hurt  existing 
relationships  with  the  community  by  supplanting  already  well-established  CBOs 
that  had  been  providing  similar  services  to  the  community.  In  general,  it  was 


13The  Results  and  Performance  Accountability  Implementation  Guide, 
Section  2.12,  “How  do  we  identify  what  works  to  improve  conditions  of  well¬ 
being?”,  http://www.raauide.ora/2  12.htm. 
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thought  that  extending  the  planning  period  might  have  helped  prevent  some  of 
these  issues  from  arising. 

The  final  area  interviewees  suggested  could  be  improved  by  additional 
planning  time  was  in  providing  evidence  for  projects’  efficacy.  The  RBDM 
Framework  urges  planners  to  use  the  best  available  evidence  in  choosing 
projects.  Such  evidence  would  include  formal  research  evidence,  knowledge  of 
the  community,  and  information  based  on  practical  experience.  The  Workgroups 
varied  in  their  approaches.  Based  on  our  interviews,  Workgroup  participants 
who  attempted  to  bring  research  evidence  to  bear  on  the  project  selection 
process  were  limited  in  their  ability  to  do  so  by  the  short  planning  time  frame.  As 
one  Workgroup  chair  commented: 

There  wasn’t  enough  time  to  allow  for  in-depth  research  as  to 
program  options.  So  a  trade-off  had  to  be  made  between  focusing 
solely  on  evidence-based  programs  versus  considering  also  other 
approaches  that  either  the  research  hadn’t  been  done  yet  or 
programs  based  on  what  line  staff  or  CBOs  felt  were  effective. 

Proposed  projects  that  were  enhancements  of  existing  programs  relied  on 
the  lead  agency’s  experience  with  that  program  as  an  indicator  of  its 
effectiveness.  Our  field  work  identified  only  one  project  that  was  based  on  an 
evidence-based,  best-practices  program  model.  Several  Workgroup  chairs 
commented  that  their  approach  was  to  try  to  ensure  that  the  proposed  set  of 
projects  put  forward  by  their  Workgroup  included  some  projects  suggested  by 
community  groups,  even  if  there  was  no  research  evidence  to  support  including 
them. 


One  of  the  reasons  the  County  wanted  to  keep  the  planning  phase 
relatively  short  was  that  it  realized  that  it  can  be  difficult  to  sustain  intensive  time 
commitments  for  extended  periods  of  time.  Even  with  just  a  two-month  project 
selection  window,  this  concern  was  in  part  borne  out.  Because  participants  in 
the  Workgroup  planning  process  varied  from  meeting  to  meeting,  several 
Workgroup  chairs  commented  that  this  made  it  challenging  to  keep  the  process 
moving  forward,  since  new  participants  wished  to  revisit  discussions  and 
decisions  previously  made.  As  a  result,  these  chairpersons  felt  that  at  times 
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discussions  at  meetings  had  to  be  condensed  to  keep  the  Workgroups  moving 
forward  with  proposal  development. 

To  the  extent  that  adopting  the  RBDM  Framework  calls  for  wide  community 
involvement  and  for  a  unified  plan,  it  brings  with  it  vigorous  debate  about  how  to 
achieve  those  goals.  In  any  collaborative  process,  there  will  be  different  opinions 
about  what  the  final  product  should  look  like  and  different  levels  of  agreement 
about  how  well  the  final  product  will  help  achieve  the  overarching  goals  that 
guided  the  planning  process.  This  is  unavoidable,  but  it  is  also  the  case  that 
involving  more  people  in  the  planning  process  and  achieving  genuine  consensus 
takes  time.  It  is  also  the  case  that  people  have  limited  amounts  of  time  and 
energy  available  for  extended  planning  efforts,  as  well  as  competing  demands  on 
their  time.  In  retrospect,  many  of  the  LTFSS  planners  believe  that  achieving  the 
goals  of  the  RBDM  Framework  planning  process  takes  longer  than  six  months, 
though  how  long  the  process  should  and  could  realistically  take  in  Los  Angeles 
County  remains  unknown. 

DISCUSSION 

How  useful  was  the  RBDM  Framework  in  planning  the  LTFSS  Plan?  The 
RBDM  Framework  served  to  focus  planners  on  ultimate  results  and  on  an  explicit 
objective  of  helping  families  to  achieve  long-term  self-sufficiency.  It  also 
encouraged  broad  community  involvement  which  appears  to  have  been 
achieved  in  the  initial  planning  process  (i.e.,  in  defining  long-term  family  self- 
sufficiency  and  in  selecting  outcome  indicators  to  measure).  However,  the 
Workgroup  planning  process  that  led  to  the  development  of  the  46  projects 
appears  to  have  been  less  successful  in  maintaining  the  high  level  of  community 
input  called  for  by  the  RBDM  Framework. 

Applying  the  RBDM  Framework  would  have  been  facilitated  by  a  longer 
planning  period.  For  example,  the  RBDM  Framework  urges  planners  to  use  the 
best  available  evidence  and  states  as  part  of  the  planning  process  it  is  important 
to  “fit  the  pieces  together"  and  to  consider  how  proposed  programs  or 
approaches  fit  together  into  a  system  of  services  and  supports,  “not  just  a  loose 
confederation  of  good  ideas”  (Freidman,  2001,  2.12).  However,  the  RBDM 
Framework  does  not  specify  a  time  frame  over  which  planning  should  be 
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accomplished,  only  that  planning  should  be  an  iterative  process,  in  which  results 
feed  back  into  additional  planning  and  efforts  to  refine  the  overall  program  and 
its  component  projects. 

In  contrast,  the  LTFSS  planning  period  was  quite  short.  Based  on  the 
County’s  previous  experience  in  planning  welfare  reform,  only  six  months  were 
allocated  for  overall  planning,  including  eight  weeks  for  developing  project 
proposals.  The  end  result  of  this  short,  intensive  planning  process  was  that  little 
time  was  available  to  discuss  other  possible  options  for  spending  the  PIF,  to 
cross-fertilize  ideas,  to  bridge  similar  concepts  between  Workgroups,  to  consider 
more  fully  the  Plan’s  service  integration  objectives  and  how  proposed  projects 
would  fit  into  existing  services  in  the  County,  and  to  bring  more  research 
evidence  and  practical  experience  to  bear  on  project  efficacy  and  cost- 
effectiveness.  In  other  words,  little  time  was  available  to  accomplish  some  of  the 
planning  steps  outlined  in  the  RBDM  Framework  and,  as  a  result,  the  usefulness 
of  the  RBDM  Framework  in  the  planning  process  was  attenuated. 
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4.  FINDINGS  ABOUT  THE  EFFECTS  OF  THE  LTFSS  PLAN 
ON  THE  IMPLEMENTATION  PROCESS 


INTRODUCTION 

In  the  previous  chapter,  we  presented  our  findings  about  the  utility  of  the 
RBDM  Framework  for  the  planning  process  that  led  up  to  the  final  LTFSS  Plan. 
In  this  chapter,  we  consider  the  Plan  and  issues  that  arose  in  implementing  the 
individual  projects  and  the  Plan’s  progress  toward  service  integration.  As  noted 
earlier,  the  RBDM  Framework  emphasizes  collaboration  and  partnerships  but  is 
relatively  silent  about  how  this  might  be  accomplished.  The  RBDM  Framework 
urges  an  overarching  plan  focused  on  achieving  population-level  outcomes.  It 
states  that  achieving  the  desired  outcomes  is  the  responsibility  of  the  community, 
that  the  community  is  accountable  for  those  results.  This  broad  effort  requires 
partners  to  work  together. 

As  we  noted  in  Chapter  2,  project  implementation  and  expenditures  are 
rolling  out  more  slowly  than  anticipated.  Some  of  the  slow  project  roll-out  is 
generic  to  the  roll-out  of  any  new  project.  However,  some  of  the  slow  project  roll¬ 
out  also  appears  to  have  been  the  result  of  the  LTFSS  Plan  itself  and  of  the 
more  complicated  program  designs  (e.g.,  flowing  funds  across  agencies, 
integrated  service  delivery,  high  levels  of  community  involvement).  Research  on 
service  integration  and  service  coordination  suggests  that  the  slow  roll-out  and 
many  of  the  implementation  challenges  we  discuss  below  are  common  to  these 
kinds  of  efforts.  Commonly  cited  barriers  (Martinson,  1999;  Holcomb  et  al., 
1993;  Burbridge  and  Nightingale,  1989;  Trutko  et  al.,  1991)  to  service 
coordination  and  integration  include: 

•  Bureaucratic  barriers  and  turf  protection  (e.g.,  differences  in 
reporting  processes  and  procedures,  concerns  about  loss  of 
decision-making  autonomy,  and  lack  of  ownership  of  projects) 

•  Different  philosophies  or  missions  (e.g.,  different  perceptions 
about  which  clients  should  be  served,  how  they  should  be 
served,  and  how  success  should  be  measured,  as  well  as 
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differences  in  the  relative  emphasis  placed  on  social  services 
versus  employment  and  training) 

•  Differences  in  performance  measures  and  in  obtaining  credit  for 
services  and  results  (e.g.,  health  services,  education,  housing, 
and  welfare  programs  are  accountable  to  different  oversight 
bodies  to  whom  they  must  provide  different  evidence  on  their 
programs’  performance;  loss  of  control  over  flow  of  clients  or 
concerns  about  meeting  numeric  service  goals  may  lead  to  a 
reluctance  to  participate  in  integrated  service  delivery) 

•  Incompatible  management  information  systems  (e.g., 
inconsistency  in  data  collection  and  management  and  different 
data  systems  and  restrictions  about  sharing  client  information) 

•  Different  eligibility  restrictions  (e.g.,  differences  in  eligibility 
restrictions  for  welfare  programs  versus  education,  health, 
training,  or  other  types  of  programs). 

Finally,  some  of  the  slow  project  roll-out  appears  to  have  been  the  result  of 
one-time  issues  related  to  developing  generic  procedures  to  implement  the  more 
complicated  project  designs  of  the  LTFSS  Plan.  For  example,  before  any  non- 
DPSS  project  could  begin  spending  funds,  DPSS  needed  to  create  procedures 
for  reviewing  project  materials.  Project  roll-out  was  delayed  while  procedures 
were  developed.  The  next  time  a  project  is  to  spend  DPSS  funds,  the  basic 
procedures  are  now  in  place  and  this  cause  of  slow  project  roll-out  should  not 
reoccur. 

THE  EFFECTS  OF  THE  LTFSS  PLAN  ON  THE  IMPLEMENTATION  PROCESS 

The  LTFSS  Plan  Has  Encouraged  County  Departments  and  Other  Lead 
Agencies  to  Work  Together  Toward  a  Common  Set  of  Goals 

In  adopting  the  RBDM  Framework,  the  LTFSS  Plan  served  to  focus  the 
County  on  achieving  a  set  of  common  goals.  A  number  of  interviewees 
commented  that  the  LTFSS  Plan  encouraged  and  facilitated  County  departments 
and  agencies  in  working  together  to  help  needy  families  in  the  County  by  shifting 
the  emphasis  from  individuals  or  specific  service  needs  to  a  focus  on  all  of  the 
needs  of  a  family  unit. 
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One  lead  agency  interviewee  commented  that  establishing  multidisciplinary 
teams  has  required  each  agency  to  conduct  business  more  flexibly  and  to  work 
together  to  determine  what  is  best  for  each  participant.  A  third  of  lead  agency 
interviewees  agreed  that  working  together  on  LTFSS  Plan  has  led  to  a  growing 
level  understanding  of,  and  respect  for,  each  other’s  programs  and  approaches 
to  working  with  clients.  For  example,  interviewees  of  one  lead  agency  discussed 
how  other  departments’  line  staffs  were  initially  wary  of  having  them  involved  in  a 
particular  program.  However,  after  extensive  training,  the  staff  learned  that  each 
of  the  participating  lead  agencies  had  similar  approaches  to  working  with  clients. 
Thus,  as  a  result  of  the  LTFSS  Plan,  program  and  line  staff  from  the  various 
agencies  became  more  comfortable  working  collaboratively  on  this  project.  This 
appears  to  be  true  even  where  there  was  resistance.  At  least  four  interviewees 
observed  that  the  LTFSS  Plan  will  “force”  agencies  to  become  familiar  with  each 
other’s  programs  so  that  they  are  better  able  to  make  appropriate  referrals  for 
these  families.  As  one  interviewee  noted,  “LTFSS  is  different  than  previous 
attempts  because  it  really  focuses  our  efforts  on  service  integration  in  a 
meaningful  way.” 

The  LTFSS  Plan  Has  Helped  to  Improve  Relationships  Between  County 
Departments  and  the  Community 

The  Plan  has  facilitated  cooperation  not  only  between  County  departments, 
but  also  between  the  departments  and  the  community.  Moreover,  for  a  number 
of  projects,  closer  relationships  between  County  departments/agencies  and  the 
community  resulted  in  closer  collaboration  on  non-LTFSS  Plan  service  delivery 
issues.  For  example,  a  side  benefit  of  one  LTFSS  project  has  been  the 
formation  of  a  Countywide  advisory  committee  of  public  and  private  home 
visitation  programs,  which  is  now  jointly  addressing  issues  broader  than  the 
LTFSS  Plan  itself. 

Another  LTFSS  project  found  that  the  community  contractors  who  conduct 
outreach  activities  have  served  as  important  advocates  for  the  County 
department’s  program  and  have  helped  to  break  down  preconceptions  within  the 
community  about  this  department.  Nor  have  the  changes  in  preconceptions 
been  one-sided.  One  LTFSS  project  helped  a  County  department  work  with  the 
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community  and,  in  the  process,  opened  up  the  way  the  department  views  its  role 
and  approach  to  those  kind  of  programs.  As  summarized  by  one  interviewee: 

The  County  system  is  not  used  to  working  with  the  community.  Only 
within  the  past  five-seven  years  have  we  realized  that  there  is 
something  else  out  there.  .  .  .  Our  LTFSS  project  has  helped  to 
overcome  past  negative  impressions  by  the  community  of  our 
department  “steamrolling”  over  community  organizations.  The 
community  is  finally  getting  to  know  and  to  trust  us. 

This  view  was  echoed  by  several  lead  agencies,  who  commented  that 
working  with  the  LTFSS  Plan  target  population  and,  in  particular,  CalWORKs 
recipients,  is  new  to  their  department.  Their  LTFSS  projects  have  enabled  them 
to  begin  developing  new  relationships  with  the  community  (as  well  as  with  other 
County  agencies).  These  cooperative  alliances,  it  should  be  noted,  are  exactly 
what  the  RBDM  Framework  was  designed  to  accomplish. 

In  addition,  the  LTFSS  Plan  encouraged  the  formation  of  both  public  and 
private  partnerships.  Table  4.1  presents  a  summary  of  relationships  within  the 
LTFSS  Plan  between  the  lead  agencies  and  other  entities  across  the  individual 
projects. 
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Table  4.1 

Coordination  Issues  among  the  LTFSS  Plan  Projects 


Project  No.  and  Name 


CalWORKs  WtW  Strate 


Career  Plan  and  Prep  Seminar 


Enhanced  Appraisal 


Tarqeted  Initial  Job  Search 


Part-Time  Work  w / 
Educ/Trainin 


Voluntary  Enhanced  Motivation 


Employer-Linked 
Educ/T  rainin 


Transitional  Subsidized 
Employment 


County  Apprenticeship 


Lead 


Provider/Supporting  Agenc 


Business  Micro-Loan  and 
Incubator 

CDC 

Business  Micro-Loan  Program 

CDC 

Incubator  Without  Walls 

CDC 

Housing  Relocation  Program 

DPSS 

Creation 


Community  Economic 
Development 


Mini-Career  Centers 


Community  Outreach — Health 
Care 


1 931  (b)  Medi-Cal  Outreach 


Prenatal  Outreach 


CalWORKs  Family  Assistance 


Media  Outreach 


Hotline  to  Resolve  Health  Care 


Health  Care  Transportation 


Health  First 


Transitional  Support 
Homeless 


Relocation  Grant — Homeless 
Families 


Tenant-based  Trans.  Rental 
Asst. 


Trans.  Subsidized  Employment 


Emergency  Assistance — 
Eviction 


Housing  Counseling/Trainin 


Community-Based  Teen 
Sen/ices 


Teens  with  Special  Needs 


Emancipated  Foster  Youth- 
Parents 


Teen  Passport  to  Success 


Staff  Development  for  Teen 
Service  Providers 


Cal-Learn  and  Teen  Parents 


Teen  Career  Enhancement 


Career  Counselors 


Status  Pending 


CDC 

Contr. 

DPSS 

Contr. 

DPSS 


DHS 


DPSS 


DPSS 
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Proiect  No.  and  Name 


Lead 


Provider/Supporting  Agenc 


Youth  Jobs 

CSS 

Public  Library  Svcs  for 

Children 

Library 

24a 

Homework  Center 

Library 

24b 

Teen  Library  Card  Campaign 

Library  | 

24c 

Support  Services  for  After- 
School  Program 

25 

Operation  READ 

Prob. 

f26 

Safe  Places 

LACOE 

27 

DART/STOP  for  CW  Families 

CSS 

28 

Domestic  Violence  Prevention 

css 

28a 

Risk  Assessment  Tool 

css 

28b 

Research:  What  Stops  DV 

css 

28c 

DV  Teen  Curriculum 

css 

29 

School-Based  Probation 
Supervision 

Prob 

30 

Support— Families  of  Probation 
Child 

Prob 

31 

Strategic  Support  for  Child 

Care 

DPSS 

31a 

Child  Care— Non-needy 
Caregiver 

DPSS 

31b 

Inc  Non-traditional  Child  Care 

DPSS 

32 

Federal  Family  Supp  Svcs 
Network 

DPSS 

33 

LTFSS  Family  Preservation 

DCFS 

34 

Home  Visitation  Program 

DHS 

34a 

Nurse  Home  Visitation  Prog 

DHS 

34b 

Home  Visitation  and  Case 
Management 

DHS 

35 

Peer  Self-Help  Support  Groups 

DMH 

36 

Support  and  Therapeutic 

Options 

37 

School  Attendance  Areas 

CAO/ 

SIB 

38 

Multi-Disciplinary  Family 
Inventory 

DPSS 

39 

County  Family  Resource 

Centers 

DPSS 

39a 

County  Family  Resource 

Centers 

DPSS 

39b 

DPOs  for  Family  Resource 
Centers 

DPSS 

40 

Data  Partnerships 

CPC 

41 

SPA  Council  Staff  and  Tech 

Asst. 

CPC 

42 

CalWORKs  Systems  Review 

DPSS 

New  Directions  L-T  Fam 

Comm 

DPSS 

44 

CalWORKs  Case  Management 

DPSS 

45 

TranStar  Enhancement 

DPSS 

46 

L-T  Family  Self-Suff  Evaluation 

CAO/ 

SIB 

Funded  from  another  source 


LACOE  LAUSD  Contr. 


DCFS  Prob  LACOE  DMH 


-55- 


There  are  1 1  lead  agencies  for  the  LTFSS  Plan,  each  assigned  individual 
projects  to  implement.  As  shown  in  the  table,  in  addition,  the  lead  agencies  also 
play  supporting  roles  to  other  projects,  including: 

•  Serving  as  service  providers  for  other  projects 

•  Co-locating  staff  with  other  departments  or  agencies  as  part  of 
an  LTFSS  project  or  having  staff  that  comprise  part  of  multi¬ 
disciplinary  teams 

•  Having  treatment  providers  who  may  be  affected  by  other 
LTFSS  projects  (e.g.,  referrals  will  be  made  to  these  providers) 

•  Co-leading  an  LTFSS  project  with  another  lead  agency 

•  Providing  technical  support  to  other  projects  (e.g.,  assistance  in 
developing  monitoring  tools). 

As  shown  in  the  table,  35  out  of  the  46  projects  and  subcomponents  have 
at  least  one  provider/supporting  agency  relationship  to  coordinate,  usually  a 
contractor.  Moreover,  12  of  the  46  have  more  than  one  provider/supporting 
agency  to  coordinate,  and  projects  38  and  39 — Multidisciplinary  Family  Inventory 
and  County  Family  Resource  Centers — have  six  and  five  relationships  to 
coordinate,  respectively.  Overall,  the  table  highlights  the  complexity  of  the 
coordination  issue. 

In  addition,  coordination  further  has  to  occur  with  other  entities  not  listed  in 
this  table,  including  the  SPAs,  the  Board,  law  enforcement  agencies,  school 
districts,  city  agencies  or  departments,  colleges,  Housing  Authorities,  private 
industry,  and  planning  bodies,  as  well  as  with  other  entities  that  either  may  be 
affected  by  LTFSS  projects  or  that  may  serve  as  contractors. 

Coordination  requires  consensus-building  and  reconciling  differences 
between  lead  agencies  in  departmental  philosophies,  in  professional  cultures, 
and  in  fundamental  views  about  the  approach  for  some  projects.  In  one  case, 
two  lead  agencies  have  faced  the  challenge  of  jointly  developing  a  project  that 
required  reconciling  fundamental  differences  in  their  programmatic  approaches 
to  this  type  of  program.  They  needed  to  come  to  an  agreement  about  how 
broadly  the  target  population  should  be  defined  and  about  the  role  of  community- 
based  providers.  As  the  agencies  noted: 
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The  biggest  challenge  in  working  with  other  agencies  is  to  “change 
their  thinking,”  to  bring  everyone  together  to  implement  a  project. 

Over  time,  we  have  seen  different  agencies  open  up  and  share  their 
views  . . .  [and]  we’ve  begun  working  toward  a  common  set  of  goals. 

There  are  more  practical  concerns  to  consider  as  well.  Co-location  of  staff 
or  forming  of  multidisciplinary  teams  requires  not  only  selecting  sites  and  forming 
teams,  but  also  addressing  such  issues  as  managerial  structures  and  the  sharing 
of  client  data.  On  a  project  where  six  lead  agencies’  personnel  comprise  the 
multi-disciplinary  teams,  we  heard  that,  “[y]ou  can  multiply  by  three  what  is 
required  to  set  up  the  administrative  and  supervisorial  structures  necessary  to 
oversee  these  teams.”  Another  interviewee  commented: 

This  project  hasn’t  moved  along  as  quickly  as  I  had  hoped.  The 
concept  was  new — more  than  just  co-location  of  staff.  There  was  a 
need  to  come  up  with  a  better  case  management  plan  that  would 
cross  bureaucratic  boundaries  and  deal  with  the  problem  of  lack  of  a 
common  database  across  departments  and  [with  the]  ability  to  share 
data  about  cases  we  now  had  in  common. 

The  Plan’s  High  Coordination  Demands,  in  Turn,  Require  Resources  for 
Coordinating  Planning  Activities  Prior  to  Service  Delivery 

One  implication  of  the  Plan’s  high  coordination  demands  is  that  someone 
has  to  pay  for  the  significant  time  and  effort  that  goes  into  building  strong 
working  partnerships  between  service  providers,  both  public  and  private. 
Overall,  the  majority  of  interviewees  noted  that  the  sheer  amount  of  coordination 
required  for  LTFSS  projects  involved  considerable  staff  time  and  administrative 
resources.  However,  the  majority  of  interviewees  also  commented  that  since 
projects  are  not  allowed  to  start  spending  their  funds  until  their  implementation 
plans  have  been  approved,  there  is  no  funding  to  support  many  of  these 
activities.  Yet  much  planning  and  coordination  must  take  place  beforehand  to 
fully  develop  programs  and  address  such  issues  as  co-location  and  cross¬ 
supervision  of  staff,  definition  of  target  populations  and  program  eligibility,  and 
selection  of  program  sites.  Further,  a  number  of  project  interviewees 
commented  that  coordination  is  required  not  simply  for  the  projects  on  which 
they  were  the  lead  agency,  but  also  for  projects  on  which  they  have  supporting 
roles.  Most  of  the  lead  agencies  did  not  expect  to  receive  reimbursement  for 
many  of  these  activities.  Some  interviewees  suggested  that  future  applications 
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of  the  RBDM  Framework  take  these  additional  “planning  for  implementation” 
costs  into  account  when  constructing  budgets. 

Applying  the  Plan  Is  Problematic  Given  the  County’s  Structure 

By  its  charter  and  by  its  practices,  the  County  has  a  Board,  no  elected 
executive,  and  strong  departments  that  report  directly  to  the  Board.  As  a  result, 
inter-department  operations  require  inter-department  negotiations — negotiations 
that  are  between  equals.  The  Board  attempted  to  address  this  issue  by 
delegating  authority  over,  and  responsibility  for,  the  LTFSS  Plan  to  the  NDTF — 
an  inter-agency  group.  However,  the  NDTF  structure  does  not  appear  to  have 
been  sufficient  to  ameliorate  inter-departmental  coordination  issues.  Thus,  it 
remained  true  that  inter-department  operations  were  negotiations  between 
equals.  As  a  consequence,  project  roll-out  was  often  delayed  by  a  drawn-out 
process  of  negotiating  cooperative  agreements.  However,  the  Plan  does  not 
account  for  this  additional  complication  (e.g.,  by  adding  additional  time  and 
resources  up  front  to  plan  for  implementation  before  projects  are  expected  to 
begin). 

Furthermore,  our  interviews  suggested  that  the  NDTF  was  unable  to 
prevent  the  LTFSS  Plan  from  being  perceived  as  a  DPSS  effort  and  that  some 
interviewees  interpreted  that  as  being  the  Board’s  intention.  To  elucidate  the 
issue,  we  turn  first  to  the  interagency  structure  of  the  County  and  then  to  how  the 
Plan  is  managed  by  the  agency  tasked  with  oversight.  Figure  4.1  shows  that  11 
different  lead  agencies  are  assigned  responsibility  for  implementing  specific 
LTFSS  projects.  The  numbers  in  parentheses  indicate  the  number  of  projects 
(out  of  the  total  46)  that  each  lead  agency  is  responsible  for.  DPSS  has,  in 
addition  to  its  projects,  a  fiscal  oversight  role  and  the  responsibility  for 
coordinating  the  implementation  of  LTFSS  projects.  In  this  capacity,  DPSS 
oversees  and  coordinates  the  review  and  clearance  process  for  Board  approval 
of  projects’  implementation  plans  and  MOUs/contracts. 
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•  Fiscal/ 

•  Lead 

•  Lead 

•  Lead 

•  Lead 

•  Lead 

*  Lead 

•  Lead 

•  Lead 

•  Lead 

prog. 

agency 

agency 

agency 

agency 

agency 

agency 

agency 

agency 

agency 

oversight 

(3/46)* 

(1/46) 

(3/46) 

(4/46) 

(3/46) 

(5/46)** 

(1/46) 

(1/46) 

(2/46) 

•  Overall 
coord  of 
LTFSS 
projects 

•  Lead  agency 
(22/46) 


•Shares  joint  lead  on  Project  10  with  DPSS 
“One  project  shifted  lead  to  DPSS 


Figure  4.1— LTFSS  Plan  Organizational  Structure 

Overall  oversight  of  LTFSS  projects’  implementation  was  assigned  to 
DPSS.  The  Board  wanted  one  department  to  be  accountable  for  how  the  money 
was  being  spent.  DPSS,  as  Chair  of  the  NDTF,  had  led  the  initial  development 
of  the  LTFSS  Plan,  and  the  SA  and  PIF  dollars  resulting  from  the  CalWORKs 
legislation  passed  through  this  department.  Thus,  several  interviewees 
explained  that  it  was  DPSS  to  whom  the  Board  would  turn  to  if  it  felt  a  project 
was  rolling  out  slowly  and  for  an  accounting  of  how  the  LTFSS  Plan  funds  were 
spent. 

To  deal  with  these  new  responsibilities  (new  LTFSS  Projects  and  the 
LTFSS  Plan  oversight  role),  DPSS,  as  part  of  the  department’s  reorganization, 
created  the  LTFSS  Plan  Division  (as  shown  in  bold  in  Figure  4.2).  This  new 
division  is  responsible  for  coordinating  all  the  LTFSS  projects  and  for  overseeing 
the  review  and  clearance  process  internally  for  LTFSS  project  documents.  It  is 
also  the  lead  on  six  of  the  twenty-two  LTFSS  projects  assigned  to  DPSS  and  has 
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responsibility  for  the  contracts  and  monitoring  function  for  a  number  of  the 
LTFSS  projects  for  which  DPSS  is  the  lead  agency. 


Figure  4.2— How  the  LTFSS  Plan  Division  Fits  Within  DPSS 


In  addition  to  the  LTFSS  Plan  Division,  the  Contracts  Management  and 
Monitoring  Division  and  the  Financial  Management  Division  have  played 
important  roles  in  the  LTFSS  Plan.  The  Contracts  Management  and  Monitoring 
Division  is  responsible  for  reviewing  relevant  LTFSS  Plan  documents  (i.e., 
implementation  plans;  requests  for  proposals,  RFPs;  and  contracts)  and  for 
overseeing  the  writing  of  contracts  for  the  LTFSS  projects  that  DPSS  has  the 
lead  on,  as  well  as  for  providing  support  to  the  LTFSS  Plan  Division. 

The  contracts  being  written  for  LTFSS  projects  represent  a  new  area  for 
DPSS.  Under  the  LTFSS  Plan,  some  RFPs  are  written  very  broadly,  giving  only 
general  guidance  about  the  type  of  programs  in  which  the  projects  are 
interested.  As  a  result,  the  proposals  submitted  by  the  community  may  cover  a 
wide  range  of  services,  for  example,  from  anger  management  classes,  to 
mentoring  programs,  to  sex  education  classes.  The  proposals  also  vary 
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considerably  in  their  level  of  specificity  about  what  services  will  be  provided  and 
how  proposed  programs  will  realize  TANF  goals. 

These  exploratory  approaches  to  service  provision  involve  the  community 
to  a  great  extent  in  shaping  the  LTFSS  Plan,  which  was  one  of  the  Plan’s  goals. 
However,  DPSS  interviewees  note  that  the  variation  in  quality  and  level  of 
specificity  has  represented  a  significant  challenge  for  DPSS  contracts  staff 
reviewing  the  proposals  and  writing  statements  of  work  to  allow  for  effective 
monitoring  of  services  delivery  and  compliance  with  the  terms  and  conditions  of 
these  contracts. 

Just  as  the  LTFSS  Plan  expanded  the  role  of  the  Contracts  Management  & 
Monitoring  Division,  it  has  also  expanded  the  role  of  the  Financial  Management 
Division.  DPSS  has  primary  fiscal  responsibility  for  LTFSS  Plan  funding, 
including  review  of  project  budgets,  implementation  plans,  RFPs,  and  contracts 
(when  needed).  The  Financial  Management  Division  is  responsible  for  the  fiscal 
oversight  of  the  LTFSS  projects,  including  the  clearance  of  LTFSS  project 
documents  with  County  Counsel  and  their  CAO  Budget  Analysts;  the  division 
also  provides  support  to  the  LTFSS  Plan  Division. 

During  the  Countywide  planning  process,  projects  only  had  to  provide  an 
estimate  of  costs,  whereas  in  the  implementation  phase,  projects  are  required  to 
submit  detailed  line-item  budgets  and  comply  with  DPSS’s  fiscal  reporting 
requirements.  However,  because  County  agencies  and  departments  vary  in  their 
fiscal  reporting  policies  and  procedures  and  in  how  their  budgets  are  set  up, 
DPSS  fiscal  staff  interviewees  commented  that  this  initially  created  some 
problems  in  terms  of  standardizing  the  development  of  project  budgets,  contracts 
and  subcontracts,  MOUs,  and  fiscal  reporting  procedures  for  LTFSS  projects. 

This  structure  had  two  major  operational  consequences  for  projects.  The 
first  is  largely  the  result  of  the  Plan’s  emphasis  on  collaboration,  which  takes 
time.  Project  roll-out  was  slowed  as  the  departments — DPSS  and  the  other  lead 
agencies — recognized  that  there  were  considerable  formal  coordination  issues 
that  needed  to  be  worked  out  and  that  procedures  needed  to  be  developed  to 
address  them.  These  procedures,  however,  had  to  conform  to  DPSS’  legal 
obligation  to  account  for  spending  federal  and  state  dollars,  which  meant  that 
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DPSS  had  a  significant  role  in  approving  projects’  implementation  plans.  This 
responsibility  was  not  obvious  at  first  to  the  other  agencies. 

Thus,  the  second  consequence  was  that  although  both  the  RBDM 
Framework  and  the  Plan  call  for  coordination  and  cooperation,  DPSS  came  to  be 
seen  as  dictating  the  terms  of  collaboration  when  it  came  to  the  details  of 
implementation.  This  was  partially  due  to  DPSS’s  need  to  impose  the  procedural 
requirements  of  the  sources  of  the  funds  and  this  was  partially  the  result  of  trying 
to  implement  the  Plan  within  the  existing  County  governmental  structure. 

DPSS  provided  some  technical  assistance.  A  first  Technical  Advisory 
Workshop  was  held  in  September  2000.  A  second  workshop  was  held  in  April 
2001  after  the  establishment  of  the  new  LTFSS  Division  in  January  2001  and  the 
formalization  of  the  review  and  clearance  process  in  the  spring  of  2001. 
Nevertheless,  projects  felt  that  initially  they  had  received  minimal  guidance  from 
DPSS  on  developing  their  implementation  plans  and  Board  letters.  In  fact,  the 
majority  of  project  interviewees  commented  that  it  was  not  apparent  at  first  that 
the  lead  agencies  would  have  to  follow  DPSS’s  detailed  requirements  in 
developing  their  Board  letters,  line-item  budgets,  and  other  components  of  their 
implementation  plans.  One  of  the  leaders  in  the  earlier  Countywide  planning 
process  commented  that,  in  retrospect,  he  would  have  involved  DPSS  contracts 
and  fiscal  staff  earlier  in  the  planning  process  so  that  they  could  start  working  on 
some  of  the  administrative  issues  that  would  need  to  be  addressed  during  the 
implementation  phase. 

Project  interviewees  also  noted  that  the  fiscal  reporting  and  budgeting 
process  was  a  challenge.  Each  of  the  lead  agencies  has  its  own  set  of 
guidelines  and  regulations  it  has  to  follow,  with  no  standardized  contracting  or 
budgeting  processes  across  County  departments.  One  third  of  project 
interviewees  commented  on  problems  associated  with  these  two  processes.  As 
summarized  by  one  interviewee: 

It  wasn’t  the  program  pieces  that  got  questioned  or  challenged,  but 
rather  the  funding  and  budget  pieces.  The  fundamental  problem  is 
that  different  funding  sources  have  different  levels  of  accountability. 

Our  funding  is  pretty  direct  and  so  the  requirements  aren’t  as 
detailed,  whereas  this  is  not  the  case  for  DPSS.  Also,  we  each  use 
different  languages.  For  example,  DPSS  refers  to  an  MOU,  whereas 
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our  department  uses  MOBs.  Further,  DPSS  has  a  large  budget  staff, 
whereas  our  administrative  staff  is  small,  with  one  individual  working 
fiscal  issues  for  all  our  projects.  DPSS  just  couldn’t  tell  us  what  to  do. 

They  were  looking  at  us  as  contractors,  not  another  County  agency. 

It  meant  that  they  were  asking  for  things  we  couldn’t  provide — e.g.,  a 
detailed  line-item  budget  that  breaks  out  employment  benefits. 

Many  of  the  projects  perceived  that  the  detailed  fiscal  reporting  and  other 
requirements  for  the  implementation  plans  were  being  imposed  on  them 
somewhat  arbitrarily  by  DPSS.  This  led  several  project  interviewees  to  comment 
that,  “this  really  was  a  DPSS  initiative.”  Two-thirds  of  project  interviewees  also 
expressed  frustration  with  the  extensive  review  and  clearance  process,  requiring 
input  from  a  number  of  different  entities  both  inside  and  outside  DPSS — steps 
that  the  department  itself  must  follow.  As  a  result,  projects  noted,  it  often  took  up 
to  two  months  before  they  received  comments  back  from  DPSS  on  their  draft 
implementation  plans,  RFPs,  or  other  documents,  with  no  guarantee  that 
reviewers  comments  would  not  contradict  each  other. 

Frustration  with  the  process  of  negotiating  cross-agency  contracts  and 
financial  reporting  requirements  was  not  one-sided.  From  DPSS’s  perspective, 
projects  did  not  understand  that  many  of  these  requirements  were  set  by  a  state 
agency  (CDSS)  and  the  federal  TANF  regulations,  not  by  DPSS  per  se.  Thus, 
DPSS  found  wide  variation  in  the  format  and  content  of  projects’  draft 
implementation  plans  and  other  documents  submitted  for  review.  This  led  to 
DPSS  holding  a  second  Technical  Advisory  Workshop  in  April  2001  to  provide 
projects  with  step-by-step  guidelines  on  developing  the  different  implementation 
documents  and  with  an  overview  of  the  review  and  clearance  process  for  the 
implementation  plans  and  for  contracts  and  MOUs. 

Once  the  new  LTFSS  Plan  Division  was  established,  the  project 
coordinators  and  liaisons  also  initiated  one-on-one  meetings  with  the  individual 
projects,  where  the  Division  brought  together  all  the  relevant  players  within 
DPSS  to  work  with  a  project  on  its  implementation  plans;  this  process  helped  to 
address  many  of  the  coordination  issues.  Indeed,  these  meetings  were  the  ones 
most  often  cited  by  project  interviewees  as  being  particularly  helpful.  In  general, 
the  LTFSS  Plan  Division  has  been  increasing  the  amount  of  technical  support  it 
provides  to  the  projects. 


-63- 


These  kinds  of  implementation  challenges  are  directly  related  to  the  Plan’s 
objective  of  coordinating  services  and  programs  across  multiple  agencies,  as 
well  as  to  the  LTFSS  Plan’s  intent  that  PIF  dollars  be  used  by  multiple  agencies 
to  provide  services  to  the  population  being  served.  Organizational  differences  in 
reporting  processes  and  procedures  must  be  addressed  so  that  funds  can  flow 
across  County  departments  and  agencies,  while  also  adhering  to  state  and 
federal  regulations  and  reporting  requirements.  This  is  a  slow,  difficult  process  in 
any  service  integration  effort.  However,  it  is  even  slower  in  the  County  because 
of  the  County’s  governmental  structure,  with  strong  departments  who  report 
directly  to  the  Board  and  no  strong  overarching  management  entity  that  can 
mediate  interdepartmental  matters.  This  structure  reinforces  the 
compartmentalization  of  different  social  services  into  different  organizations. 
(Separate  funding  streams  with  separate  reporting  requirements  have  a  similar 
effect.)  In  turn,  this  governmental  structure  can  contribute  to  the  sense  that  one 
agency  in  its  position  as  “first  among  equals,”  imposing  its  will  on  the  others.  We 
note  that  this  is  a  potential  risk  to  any  effort  to  integrate  or  coordinate  services  in 
the  County,  regardless  of  whether  the  oversight  role  falls  to  DPSS  or  to  another 
agency.  It  is  partially  the  governmental  structure  itself  that  makes  the  RBDM 
Framework  difficult  to  implement  in  the  County. 

In  Large  Counties,  More  Formal  Links  Between  Planning  and 
Implementation  Are  Needed 

Individuals  who  developed  the  LTFSS  project  proposals  were  not 
necessarily  the  same  ones  responsible  for  implementing  the  projects,  leaving 
room  for  differences  in  interpreting  the  proposed  projects.  This  frustrated  some 
planners,  who  thought  that  their  vision  for  the  project  was  being  altered;  it  also 
frustrated  some  implementers  who  thought  that  they  did  not  necessarily  have  a 
good  understanding  of  the  planners’  intent  in  designing  a  project.  Several 
Workgroup  participants  remarked  that  detailed  project  proposals  were  briefly 
summarized  in  the  LTFSS  Plan.  The  initial  proposals  submitted  by  the 
Workgroups  during  the  Countywide  planning  process  contained  more  detailed 
program  descriptions  in  some  cases  than  the  summary  descriptions  that  made  it 
into  the  final  LTFSS  Plan. 
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ln  addition,  County  staff  charged  with  implementation  thought  that  their 
expertise  was  being  overlooked  with  regard  to  project  content  or  with  regard  to 
how  a  project  fit  into  the  service  delivery  system.  Six  project  interviewees  stated 
that  they  would  have  constructed  specific  LTFSS  projects  differently  or  had  ideas 
about  how  they  might  be  improved  but  did  not  feel  they  had  the  authority  to 
change  the  scope  of  a  project.14  Another  interviewee  from  a  lead  agency  noted 
that  his  department  already  had  an  existing  program  model  and  experience  in 
knowing  what  works  and  what  does  not;  but  the  LTFSS  project  takes  a  different 
approach.  As  a  result,  projects  can  sometimes  be  perceived  as  imposed  from 
without.  One  observer  commented  that  his  organization  was  not  involved  in 
developing  two  of  the  projects  but  now  had  responsibility  for  “implementing 
someone  else’s  vision,”  adding  that:  “If  the  lead  agency  can  see  a  better  way  to 
run  a  program  that  wasn’t  in  the  original  Plan,  there  needs  to  be  some  flexibility 
to  make  these  adjustments.”  These  tensions  were  apparent  not  only  when 
planners  were  not  County  employees,  and,  thus,  were  unfamiliar  with  County 
practices  and  capabilities,  but  also  when  representatives  from  one  agency  were 
planning  for  projects  that  would  be  housed  in  another  agency. 

One  of  the  advantages  of  using  the  RBDM  Framework  in  developing  the 
LTFSS  Plan  is  that  it  helps  bring  new  voices  and  new  perspectives  to  the  table. 
However,  this  can  also  mean  that  plans  can  be  heavily  influenced  during  a 
consensus-building  Workgroup  process  by  individuals  who  may  be  unfamiliar 
with  particular  County  departments  and  their  various  programs.  Four  LTFSS 
Plan  project  interviewees  thought  that  planners  did  not  necessarily  understand 
what  a  department’s  capabilities,  expertise,  or  priorities  were.  Another 
commented  that  his  department  does  not  necessarily  have  the  staff  or  resources 
to  provide  these  types  of  programs,  nor  much  experience  in  working  with  the 
particular  population  being  addressed  in  the  program. 

This  is  not  just  a  tension  between  community  representatives  and  the 
County  agencies.  There  were  also  differences  in  views  between  the  lead 


14Although  there  are  procedures  by  which  a  project's  implementation  plan 
can  be  modified,  the  perception  of  these  interviewees  was  that  it  was  difficult  to 
make  such  changes. 
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agencies  and  DPSS  about  projects’  foci  and  about  how  broadly  target 
populations  should  be  defined.  Underlying  these  differences  are  fundamental 
differences  between  departments  in  organizational  philosophies  and  in 
approaches  to  programs  and  service  provision.  For  example,  on  several 
projects,  DPSS  and  the  lead  agencies  disagreed  over  how  broadly  a  project’s 
target  population  should  be  defined.  In  one  case,  DPSS  wanted  to  verify  that  all 
eligible  CalWORKs  recipients  had  been  referred  before  broadening  the  definition 
of  the  target  population.  The  lead  agency,  however,  felt  that  staying  with  the 
target  population  defined  in  the  LTFSS  planning  process  would  limit  the 
recruitment  of  program  participants.  In  another  case,  the  lead  agency  felt  that  by 
staying  with  the  definition  of  the  target  population  developed  during  the  planning 
process,  the  program  would  omit  the  age  group  most  likely  to  benefit  from  this 
program’s  services. 

In  Chapter  3,  we  noted  that  the  early  implementation  stage  can  be  equally 
frustrating  for  those  who  participated  in  planning  if  they  believe  that  their  original 
vision  has  not  always  been  realized.  For  future  initiatives,  this  is  an  area  in 
which  a  longer  planning  phase  and  a  more  formal  carryover  of  the  planning 
Workgroups  into  the  early  phase  of  implementation  might  be  a  useful  strategy. 
Departmental  or  project  representatives  could  work  in  tandem  with  the  planning 
Workgroup  in  the  planning  and  early  phases  of  implementation  to  allow  for 
project  concerns  to  be  aired  and  for  planners  to  clarify  their  vision  and  amend 
the  project  as  necessary  on  an  ongoing  basis.  This,  in  turn,  could  increase  the 
departments’  sense  of  “ownership”  of,  and  thus  commitment  to,  the  projects,  and 
Workgroup  oversight  could  ensure  that  their  collaborative  vision  was  realized. 

The  LTFSS  Plan  Made  Partial  Progress  Toward  an  Integrated  Health  and 
Human  Services  Delivery  System 

The  RBDM  Framework  states  as  part  of  that  process  it  is  important  to  “fit 
the  pieces  together”  and  to  consider  how  proposed  programs  or  approaches  fit 
together  (i.e.,  with  other  projects  in  the  LTFSS  Plan  and  with  existing  programs) 
into  a  system  of  services  and  supports,  “not  just  a  loose  confederation  of  good 
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ideas.”15  Given  the  rapid  pace  of  the  Countywide  planning  process  (which  was 
discussed  in  Chapter  3),  the  individual  Workgroups  had  a  limited  amount  of  time 
to  consider  how  the  various  projects  might  be  related  or  could  be  restructured  to 
fully  integrate  the  various  concepts  and  projects.  Those  involved  in  the 
Workgroup  planning  process  felt  that  the  short  planning  time  frame  did  not  allow 
for  full  consideration  of  the  Plan’s  service  integration  objectives. 

Planners  were  not  the  only  ones  concerned  about  projects  being 
insufficiently  integrated  with  the  existing  service  delivery  system;  implementers 
raised  similar  issues.  In  at  least  two  instances,  interviewees  reported  that 
projects  overlap  with  already  existing  programs,  meaning  that  the  implementers 
must  determine  how  to  ensure  that  these  LTFSS  projects  complement  rather 
than  duplicate  existing  programs.  For  example,  three  interviewees  raised  the 
issue  of  LTFSS  projects  increasing  referrals  to  existing  sen/ice  delivery  systems 
that  already  may  be  operating  at  capacity.  They  noted  that  the  intent  of  a 
number  of  LTFSS  projects  was  to  improve  service  integration  by  being  able  to 
refer  families  to  a  range  of  programs  within  the  County  to  more  comprehensively 
address  their  service  needs.  However,  some  providers  (e.g.,  substance  abuse 
and  mental  health  providers  in  some  areas)  are  already  dealing  with  a  greater 
demand  for  services  than  there  is  a  supply.  Further,  although  referrals  can  be 
made  to  other  programs  within  the  County,  it  may  still  take  a  participant  four-six 
weeks  to  get  an  appointment  with  a  treatment  provider  or  eligibility  worker. 

Six  interviewees  also  raised  the  question  of  the  degree  to  which  County 
departments’  line  staff  are  aware  of  LTFSS  projects  and,  thus,  able  to  make 
appropriate  referrals  to  these  programs.  For  one  project,  DPSS  field  offices 
were  not  aware  of  an  LTFSS  Plan  program  and,  thus,  were  unable  to  make 
referrals  to  them;  in  another  instance,  staff  of  another  DPSS  program  were 
concerned  that  a  particular  LTFSS  project  might  compete  with  theirs  for  clients. 

In  future  applications  of  the  RBDM  Framework,  more  planning  time  would 
help  address  some  of  these  concerns.  Of  course,  it  is  also  the  case  that  people 


15The  Results  and  Performance  Accountability  Implementation  Guide. 
Section  2.12. 
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have  limited  amounts  of  time  and  energy  available  for  extended  planning  efforts, 
and  competing  demands  on  their  time.  In  retrospect,  many  of  the  LTFSS 
planners  believe  that  achieving  the  goals  of  the  RBDM  Framework  planning 
process  takes  longer  than  six  months,  though  how  long  the  process  should  and 
could  realistically  take  in  Los  Angeles  County  remains  unknown.  Finally, 
ongoing  discussions  between  Workgroup  planners  and  project  implementers 
could  also  help  in  this  area,  by  anticipating  service  bottlenecks  and  by 
disseminating  knowledge  about  existing  services. 

Adopting  the  LTFSS  Plan  Has  Led  to  Specific  Initiatives  That  Extend 
Beyond  the  Plan  Itself 

The  LTFSS  Plan  has  led  to  specific  initiatives  that  extend  beyond  the  Plan 
and  that  promise  to  yield  important  long-term  benefits  for  the  County  and  its  goal 
of  applying  the  RBDM  Framework  Countywide.  For  instance,  the  LTFSS  Plan, 
along  with  welfare  reform,  has  led  to  a  statewide  re-examination  of  contracting 
requirements  through  a  workgroup  process  involving  all  the  counties  and  the 
County  Welfare  Directors  Association  (CWDA)  to  streamline  those  procedures 
that  may  not  be  legally  mandated. 

The  Interagency  Operations  Group  (IOG)  also  has  recently  begun 
addressing  as  a  Countywide  issue  the  objective  of  incorporating  community 
relationship-building  as  an  important  consideration  in  decisions  about  rental 
space  and  the  location  of  program  sites.  One  interviewee  commented  that  this 
decision  was  partly  the  direct  result  of  projects’  experience  under  the  LTFSS 
Plan. 


Further,  the  LTFSS  Plan  is  viewed  by  many  as  being  a  test  case  to  assess 
the  feasibility  of  applying  the  RBDM  Framework  in  the  County.  As  per  CAO/SIB 
staff,  the  LTFSS  Plan  represents  an  important  opportunity  to  learn  what 
adjustments  may  be  needed  in  applying  the  RBDM  Framework  and  which  areas 
in  the  RBDM  Framework  should  be  concentrated  on.  As  a  result  of  the 
perceived  success  of  the  use  of  RBDM  Framework  in  the  development  of  the 
LTFSS  Plan,  the  RBDM  Framework  and  its  focus  on  results  and  performance 
accountability  is  now  being  expanded  to  other  areas  of  the  County  beyond  the 
LTFSS  Plan  itself.  For  example,  the  decision  was  recently  made  to  incorporate 
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the  RBDM  Framework  into  the  County’s  budget  process,  with  the  CAO  currently 
moving  forward  with  phased  implementation  of  a  plan  to  restructure  the 
children’s  budget  beginning  in  FY  2002-2003. 

DISCUSSION 

This  chapter  has  considered  the  effects  of  the  LTFSS  Plan  on 
implementation.  Compared  to  individual  projects  reviewed  and  approved  directly 
by  the  Board,  the  Plan  required  more  inter-project  and  interdepartmental 
interaction.  Projects  were  developed  by  a  public  process,  selected  by  the  NDTF 
(a  cross-departmental  group),  approved  by  the  Board  collectively,  and,  in  many 
cases,  then  approved  again  individually  by  the  Board.  Furthermore,  the  Plan 
provided  that  funds  received  by  DPSS  from  the  state  would  be  spent  by  other 
County  departments. 

Some  aspects  of  the  LTFSS  Plan  and  the  resulting  inter-project  and 
interdepartmental  interaction  facilitated  implementation  and  County  processes 
more  generally.  The  Plan  has  encouraged  County  departments  and  other  lead 
agencies  to  work  together  toward  a  common  set  of  goals  for  providing  services  to 
these  children  and  families.  The  LTFSS  Plan  also  helped  to  facilitate 
cooperation  between  County  departments  and  the  community,  and,  in  several 
instances,  cooperation  on  LTFSS  projects  led  to  improved  relationships  on 
issues  broader  than  the  LTFSS  Plan,  itself. 

Other  aspects  of  the  LTFSS  Plan  and  the  resulting  inter-project  and 
interdepartmental  interaction  hindered  implementation.  The  high  coordination 
demands  associated  with  the  LTFSS  Plan  meant  that  a  number  of  the  planning 
and  coordination  activities  necessary  to  implement  the  projects  were 
unanticipated  and  funding  and  other  resources  (e.g.,  staff)  to  perform  those 
tasks  were  often  not  available  when  they  would  have  been  most  useful.  The  lack 
of  formal  links  in  the  LTFSS  Plan  between  those  responsible  for  planning  this 
initiative  and  those  responsible  for  implementing  it  led  to  concerns  by  some 
planners  that  their  vision  for  a  project  was  being  altered  and  to  concerns  by 
some  implementers  that  they  did  not  necessarily  have  a  good  understanding  of 
the  planners’  intent  in  designing  a  project,  or  that  that  intent  was  misguided.  The 
short  planning  horizon  for  developing  the  Plan  and  the  individual  projects  did  not 
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allow  sufficient  time  for  considering  how  to  integrate  LTFSS  projects  with  existing 
service  delivery  systems. 

Finally,  applying  the  LTFSS  Plan  was  problematic  given  the  County’s 
organizational  structure  and  the  limited  amount  of  previous  inter-departmental 
cooperation.  As  a  result,  it  took  time  to  set  up  mechanisms  to  allow  for  the  flow 
of  PIF  monies  across  agencies  and  to  formalize  procedures  such  as  the  review 
and  clearance  process  for  projects.  In  conclusion,  the  overall  effect  of  the 
LTFSS  Plan  and  its  implementation  in  Los  Angeles  County  has  been  that  project 
roll-out  was  slower  than  might  have  been  the  case  in  the  absence  of  the  LTFSS 
Plan,  or  if  County  departments  had  had  more  experience  with  inter-departmental 
cooperation.  Through  the  implementation  of  the  LTFSS  Plan,  considerable 
experience  has  accumulated.  This  experience  should  ease  the  roll-out  of  future 
inter-departmental  efforts. 
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5.  FINDINGS  ABOUT  THE  UTILITY  OF  THE  RBDM  FRAMEWORK  IN  THE 

EVALUATION  PROCESS 


INTRODUCTION 

Evaluation  is  a  key  component  of  the  RBDM  Framework,  because  it 
generates  estimates  of  the  “results”  that  are  then  used  to  refine  the 
implementation  and  to  guide  future  project  decisions  and  budgeting.  As 
Friedman  notes: 

Performance  budgeting  can  present  better  choices  by  requiring  each 
budget  unit  (internal  and  contract)  to  answer  the  basic  questions  in 
performance  accountability:  Who  are  your  customers?  How  do  you 
measure  if  your  customers  are  better  off?  How  do  you  measure  if 
you’re  delivering  service  well?  .  .  .  These  questions  should  be 
answered  on  a  regular  basis  throughout  the  year,  and  used  once  a 
year  to  drive  the  budget  (Friedman,  2001 ,  Section  3.16). 

Consistent  with  this  role  of  evaluation  in  the  RBDM  Framework,  evaluation 
plays  a  prominent  role  in  the  LTFSS  Plan,  both  in  planning  and  in 
implementation.  With  some  notable  exceptions  (e.g.,  the  Manpower 
Demonstration  Research  Corporation’s,  MDRC’s,  evaluations  of  the  County’s 
GAIN  program),  evaluation  has  not  traditionally  been  a  component  of  program 
roll-out  in  the  County.  Thus,  the  fact  that  there  is  a  strong  evaluation  component 
is  itself  a  reflection  of  the  positive  effect  of  the  RBDM  Framework.  Participants  in 
the  Countywide  planning  process  and  in  the  Evaluation  Design  Workgroup 
process  stated  that  the  RBDM  Framework  focused  County  departments  and 
other  key  stakeholders  on  outcomes  and  on  accountability.  Furthermore, 
interviewees  explained  that  the  RBDM  Framework  and  the  materials  produced 
by  the  CAO/SIB  helped  County  departments  to  think  more  formally  about 
program  outcomes  and  has  served  as  a  useful  framework  for  prioritizing 
departmental  resources. 

During  the  planning  process,  projects  were  to  be  selected  based  on  prior 
research  and  evaluations  whenever  possible.  In  the  implementation  phase,  an 
outside  evaluator— RAND — is  responsible  for  conducting  a  Countywide 
evaluation  of  the  impact  of  the  LTFSS  Plan  on  designated  indicators,  such  as 
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infant  mortality  and  poverty  rates,  and  to  describe  the  utility  of  the  RBDM 
Framework  in  developing  and  evaluating  the  LTFSS  Plan.  This  report  is  one  of 
the  products  of  that  ongoing  analysis.  In  addition,  each  project  is  required  to 
develop  an  evaluation  report  in  accordance  with  the  LTFSS  Plan  Evaluation 
Design  to  assess  program  impact. 

In  this  chapter,  we  provide  a  brief  overview  of  the  evaluation  process.  We 
then  consider  the  extent  to  which  those  involved  in  the  evaluation  felt  that  the 
RBDM  Framework  was  useful. 

EVALUATION  TASKS 

As  specified  in  the  County’s  LTFSS  Evaluation  Design,  project-level 
evaluations  are  to  include  the  following  steps,  as  described  below. 

Logic  Model.  Each  project  is  to  specify  the  theoretical  relationship 
between  the  project’s  intervention  and  the  outcomes  for  children  and  families. 

Performance  Measures.  Projects  need  to  articulate  performance 
measures,  against  which  to  evaluate  their  impact.  Those  performance  measures 
are  classified  according  to  the  four  quadrants  depicted  in  Table  5.1.  Project 
performance  measures  should  be  both  at  the  level  of  project  input  or  effort 
(Quadrants  I  and  II)  and  at  the  level  of  client  effects  (Quadrants  III  and  IV). 


Table  5.1 

Four  Quadrant  Schema 


Quantity 

Quality 

Input  (Process  or 
service  delivered)/ 

Effort 

Quadrant  1 

How  much  service  did  we 
deliver? 

Quadrant  II 

How  well  did  we  deliver 
the  service? 

Output  (Product  of 
client  condition 
achieved)/ 

Effect 

Quadrant  III 

How  much  effect/change 
did  we  produce? 

Quadrant  IV 

What  quality  of 
effect/change  did  we 
produce? 
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Data  Sources  Document.  Having  identified  the  performance  measures, 
the  project  then  develops  a  document  describing  the  sources  and  methods  to  be 
used  to  collect  this  data  over  time,  both  before  and  after  project  implementation. 
If  historical  data  are  not  available,  the  project  is  to  construct  a  comparable  group 
who  do  not  receive  services  to  compare  outcomes  to  project  participants. 

Progress  Graph.  From  the  identified  data  sources,  the  project  should 
tabulate  and  plot  the  historical  data  for  the  headline  performance  measure.  The 
project  should  then  use  the  historical  trend  to  develop  a  forecast  of  the  future 
level  of  the  headline  indicator  “in  the  absence  of  the  project.”  As  much  as 
possible,  the  forecast  should  include  information  on  other  factors  that  may  affect 
the  headline  performance  measure. 

Stories  Behind  the  Baseline.  A  project  then  constructs  a  narrative 
discussing  the  factors  likely  to  affect  the  baseline.  What  explains  differences  in 
outcomes  across  subgroups?  What  contributed  to  the  historical  trend?  What 
factors  influenced  the  forecast’s  development?  What  factors  may  have 
influenced  actual  trends  after  program  implementation?  What  differences  exist 
between  baseline  and  actual  data  trendlines,  and  why  do  they  differ? 

Identifying  Partners.  The  RBDM  Framework  emphasizes  that  outcomes 
do  not  change  because  of  government  efforts  alone.  Each  project  is  asked  to 
identify  other  organizations — public  and  private  partners — that  have  affected  or 
could  affect  the  outcomes  of  interest. 

Quality  Improvement  Steps.  Having  considered  outcomes  to  date, 
projects  should  consider  what  changes  can  be  made  to  improve  outcomes.  This 
is  the  key  step  in  continuous  quality  improvement. 

UTILITY  OF  THE  RBDM  FRAMEWORK  IN  THE  EVALUATION  PROCESS 

The  LTFSS  Plan  Evaluation  Design  Helped  Introduce  Lead  Agency  Staff  to 
the  RBDM  Framework 

Under  the  clear  influence  of  Friedman,  the  LTFSS  Plan  emphasizes  the 
importance  of  focusing  on  outcomes  and  on  measuring  progress  toward  those 
outcomes.  Consistent  with  the  emphasis  on  outcomes,  every  project  has  been 
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required  to  develop  an  evaluation  that  conforms  to  an  explicitly  specified  version 
of  the  Friedman  approach.  A  number  of  participants  praised  Friedman’s  model 
for  how  it  focuses  individuals  at  all  levels  on  client-level  outcomes,  rather  than 
simply  on  process  measures.  This  focus  is,  in  several  ways,  constructive.  First, 
it  is  a  noteworthy  accomplishment  that  the  model  has  served  to  guide  County 
departments  and  agencies  in  thinking  more  formally  about  program  outcomes. 
Second,  it  has  also  allowed  them  to  focus  on  a  common  set  of  outcomes.  Third, 
it  has  also  provided  a  useful  framework  for  prioritizing  departmental  resources. 
And  fourth,  participants  felt  that  the  Friedman  model  has  helped  County 
departments  and  agencies  to  learn  to  “speak  a  common  language”  focused 
around  strategic  planning,  outcomes,  and  evaluation.  These  were  goals  of  the 
LTFSS  Plan  and  are  important  steps  toward  the  County’s  goal  to  stabilize 
families  by  building  their  capacity  to  become  self-sufficient. 

Consultations  with  Friedman  on  the  RBDM  Framework  were  described  by 
four  interviewees  as  having  made  evaluation  more  accessible  to  senior 
managers  and  to  staff  without  research  training.  As  one  senior  manager  noted, 
“when  Friedman  spoke,  it  represented  a  complete  turn  around  by  senior 
management  in  that  there  was  a  sense  of  ‘Now  I  get  it. .  .  .  ’  An  understanding 
that  evaluation  can  be  done  by  regular  people.”  Another  lead  agency 
interviewee  noted  that  evaluation  is  new  to  their  project  staff  and  Friedman’s 
training  sessions  have  helped  them  to  think  about  evaluation  in  a  more 
straightforward  way. 

Training  on  Applying  the  Evaluation  Design  Has  Been  Helpful,  but 
Interviewees  Feel  More  One-on-One  Training  Is  Needed 

In  addition  to  the  consultation  by  Friedman  himself,  the  CAO/SIB  has 
provided  other  training  and  technical  assistance.  In  December  2000,  the 
CAO/SIB  held  an  all-day  training  session  with  over  100  County  departmental 
representatives.  The  training  was  intended  to  provide  them  with  an  overview  of 
the  project  evaluation  implementation  plan  and  to  introduce  them  to  the 
Evaluation  Design  itself  and  Friedman's  approach.  CAO/SIB  prepared  and 
distributed  a  detailed  Project  Evaluation  Implementation  Guide  to  help  projects  in 
developing  project  evaluation  deliverables.  The  Project  Evaluation 
Implementation  Guide  provides  projects  with  step-by-step  guidelines  for 
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developing  their  own  evaluation,  and  sources  for  additional  references  that 
explain  the  Framework  and  its  application. 

In  recognition  of  the  challenges  associated  with  developing  the  project 
evaluations,  since  July  2000,  CAO/SIB  also  has  provided  one-on-one  assistance 
to  projects  to  help  them  develop  their  logic  models,  to  refine  their  performance 
measures,  and  to  work  through  the  other  evaluation  deliverables.  This  includes 
helping  projects  identify  best-practice  program  models  and  models  of  change 
that  might  serve  as  a  theoretical  basis  for  their  programs,  as  well  as  examples  of 
programs  that  have  been  implemented.  In  addition,  CAO/SIB  is  helping  projects 
to  identify  appropriate  comparison  groups  and  data  sources,  and  provide 
technical  assistance  to  projects  in  the  area  of  information  systems  development. 

Finally,  the  CAO/SIB  meets  at  least  quarterly  with  the  Evaluation  Design 
Workgroup.  These  meetings  bring  together  representatives  from  all  the  lead 
County  agencies.  The  Workgroup  is  charged  with  continuing  to  guide  the  design 
and  implementation  of  the  LTFSS  Plan  Evaluation  and  to  oversee  Countywide 
evaluation  deliverables.16  Interviewees  indicate  that  these  meetings  also 
represent  a  chance  to  share  progress  on  their  projects'  evaluation  deliverables 
and  to  discuss  problems  encountered  and  how  they  are  being  addressed. 

Project  interviewees  reported  that  the  training  provided  them  with  a  good 
introduction  to  the  RBDM  Framework  and  the  LTFSS  Plan  Evaluation  Design. 
However,  the  majority  of  the  interviewees  felt  they  needed  more  one-on-one 
assistance  in  developing  their  logic  models  and  in  working  through  the  steps  of 
the  projects’  evaluation  development  process.  Part  of  the  reason  for  the 
perceived  lack  of  sufficient  training  was  that  project  coordinators  and  program 
staff  varied  in  the  amount  of  training  they  had  received,  either  because  of  project 
or  lead  agency  staff  turnover  or  because  of  schedule  conflicts.  In  addition,  as 
noted  by  CAO/SIB,  at  some  of  the  earlier  sessions,  some  projects  were  not  far 
enough  along  to  fully  benefit  from  the  training.  Finally,  interviewees  from  one 
lead  agency  anticipated  a  future  challenge  as  providing  training  for  project 


16County  of  Los  Angeles  Long-Term  Family  Self-Sufficiency  Plan 
Evaluation  Design,  page  3,  October  23,  2000. 
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contractors  on  the  Evaluation  Design  to  ensure  that  they  implement  it 
appropriately. 

Projects  Have  Experienced  Various  Difficulties  in  Applying  the  Evaluation 
Design 

While  other  evaluation  approaches  focus  only  on  services  delivered  (input) 
without  considering  their  effect  on  outcomes,  Friedman’s  approach  emphasizes 
both  input  and  outcome  measures.  Involving  project  staff  in  the  effort  to 
measure  services  delivered  and  client-level  outcomes  is  also  important.  Careful 
measurement — over  a  long  period  of  time,  with  detail  on  geography  and 
background  characteristics — is  the  foundation  of  any  evaluation  effort. 

The  RBDM  Framework  itself  is  seen  by  many  participants  as  valuable. 
Nevertheless,  in  our  interviews,  interviewees  expressed  concerns  that  the  RBDM 
Framework  itself  was  difficult  to  implement  and,  thus,  that  individual  projects 
might  implement  the  Framework  differently.  This  difficulty  was  recognized  early, 
as  was  evident  in  the  early  discussions  of  the  Evaluation  Design  Workgroup.  As 
summarized  by  several  participants,  the  nature  of  those  Workgroup  discussions 
centered  on  how  to  apply  the  RBDM  Framework  and  how  to  make  it  work.  Their 
discussions  included  how  to  conceptualize  the  relationship  between  the 
individual  project  evaluations  and  that  of  the  Plan  as  a  whole,  as  well  as  the 
timing  of  the  different  elements  of  the  Evaluation  Design. 

More  generally,  interviews  with  the  members  of  the  Evaluation  Design 
Panel  indicated  three  areas  where  projects  were  having  difficulty  applying  the 
evaluation  component  of  the  RBDM  Framework:  (1)  logic  models  that  were 
missing  a  theoretical  basis  or  a  model  of  change  for  the  proposed  project;  (2) 
confusion  about  which  quadrant  a  performance  measure  may  fall  in  and,  in  some 
instances,  duplication  between  the  headline  and  secondary  measures;  and  (3) 
the  fact  that  a  few  projects  identified  more  performance  measures  per  quadrant 
that  the  Panel  felt  was  unrealistic. 

As  a  result,  a  number  of  draft  project  evaluation  formative  deliverables 
submitted  initially  were  returned  to  the  lead  agencies  for  revisions,  with  some 
requiring  multiple  revisions.  The  time  line  for  when  deliverables  were  due  was 
deliberately  set  by  the  Evaluation  Design  Workgroup  to  allow  for  the  early 
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identification  of  problems  that  projects  may  be  experiencing.  As  part  of  the 
review  process,  Panel  members  and  the  CAO/SIB  provided  a  variety  of 
suggestions  to  projects  on  how  to  strengthen  their  evaluation  formative 
deliverables,  including  possible  data  sources,  relevant  comparison  groups  or 
historical  controls,  and  additional  performance  measures  to  consider.  Several 
interviewees  stated  that  the  Panel  members  thought  it  was  important  to  address 
design  problems  early  on  rather  than  have  projects  implement  flawed 
evaluations.  This  is  consistent  with  the  emphasis  the  RBDM  Framework  places 
on  evaluation.  Evaluation  Design  Panel  members  also  noted  that  projects  are 
making  progress  in  this  area.  In  their  view,  projects  appear  to  be  improving  in 
such  areas  as  identifying  appropriate  performance  measures  in  identifying  a 
theoretical  basis  for  their  programs. 

Interviewees  commented  that  the  Evaluation  Design  Workgroup  meetings 
were  only  somewhat  helpful  to  them  in  working  through  the  development  of  their 
project  evaluation  deliverables.  Several  interviewees  commented  that  staff 
turnover  among  the  lead  agencies  has  meant  that  issues  discussed  or  decisions 
made  in  previous  meetings  had  to  be  repeated  with  new  project  staff  at 
subsequent  meetings.  They  also  commented  that  the  individuals  the  lead 
agencies  sent  to  these  meetings  varied  in  level  of  expertise  (e.g.,  data  staff  and 
program  staff),  which  led  to  some  unevenness  in  the  experience  and  focus  of 
Workgroup  discussions.  Several  interviewees  felt  that  CAO/SIB  staff  clearly 
understood  the  evaluation  concepts  and  RBDM  Framework,  but  that  they  had 
not  yet  explained  it  well  to  the  projects. 

The  CAO  LTFSS  Plan  Evaluation  Project’s  budget  includes  a  supplemental 
funding  set-aside  that  the  lead  agencies  can  request  support  for  their  project 
evaluations.  Based  on  input  from  the  lead  agencies,  the  Evaluation  Design 
Panel  identified  five  priority  areas  for  the  use  of  these  funds:  (1)  purchase 
computer  hardware  or  software;  (2)  train  staff  on  the  use  statistical  or  database 
software;  (3)  hire  evaluation  or  information  science  (IS)  consultants;  (4)  cover 
evaluation  reports’  printing  costs;  and  (5)  pay  for  staff  time  to  retrieve  data  from 
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DPSS  information  systems.17  Given  the  projects’  slow  roll-out,  project 
interviewees  and  CAO/SIB  expect  that  the  amount  of  supplemental  funding 
requested  for  project  evaluations  will  rise  in  the  latter  half  of  the  current  FY  and 
the  beginning  of  next  FY  (FY  2002-2003)  as  the  evaluation-related  technical 
assistance  needs  of  lead  agencies  rise. 

The  projects’  requests  for  supplemental  funds  to  support  the  technological 
infrastructure  required  to  track  data  do  not  solve  all  their  data  concerns. 
Members  of  the  Evaluation  Design  Panel  indicate  that  one  of  the  major  areas  in 
which  projects  are  having  difficulties  in  applying  the  RBDM  Framework  is  data- 
related.  The  Panel  noted  two  key  data-related  issues:  (1)  difficulties  in 
identifying  data  on  pre-program  outcomes  or  outcomes  for  comparison  groups; 
and  (2)  difficulties  in  identifying  data  sources  and  a  lack  of  knowledge  about  what 
data  might  be  available.  About  half  of  project  interviewees  stated  that  they  were 
not  far  enough  along  in  their  Evaluation  Designs  to  address  data-related  issues. 
Still,  a  review  of  projects’  draft  Evaluation  Designs  and  data  sources  documents 
indicate  that  projects  have  also  recognized  a  number  of  data  quality  issues  that 
may  ultimately  affect  their  ability  to  accurately  measure  performance  measures 
and  track  service  delivery. 

Data  quality  issues  may  seem  like  minor  technical  issues,  but  the  RBDM 
Framework  places  great  emphasis  on  using  data  to  inform  fiscal  and  operational 
decisions.  It  is  useful  as  something  other  than  a  planning  tool  to  the  extent  it  is 
implementable;  thus,  the  issue  of  data  quality  and  availability  (as  well  as  the 
availability  of  skills  to  understand  and  interpret  data,  as  we  discuss  below)  speak 
directly  to  the  utility  of  the  RBDM  Framework. 

Not  all  data-related  problems  are  directly  RBDM  Framework-related. 
Issues  related  to  confidentiality  restrictions,  sharing  of  client  information,  and 
sharing  of  data  (e.g.,  no  consistent,  unique  identifiers  across  departmental 
databases)  also  have  been  experienced  by  the  State  and  other  counties  in  the 
implementation  of  welfare  reform  and  CalWORKs,  as  well  as  the  LTFSS  Plan 


17Memo  entitled  “LTFSS  Plan  Evaluation  Supplemental  Resources”,  sent 
to  LTFSS  Coordinators,  from  A.  Drakodaidis,  CAO/SIB,  May  7,  2001 . 
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(Klerman  et  al.,  2001).  Other  problems  are  a  direct  implication  of  moving  toward 
results-based  accountability,  not  just  in  the  County  but  across  the  country. 
Research  on  applying  results-based  decision  making  and  budgeting  approaches 
indicates  that  an  important  challenge  facing  funders  and  administrators  is 
establishing  the  infrastructure  necessary  to  collect  and  analyze  outcomes  data 
(Liner  et  al.,  2001).  This  is  true  not  just  for  County  entities  but  also  for 
contractors  and  community  participants.  Outcome  measurement  is  relatively 
new  to  many  private  nonprofit  organizations,  who  are  used  to  only  monitoring 
and  reporting  such  information  as  the  number  of  clients  served  and  the  quantity 
of  services,  programs,  or  activities  provided  (Morley  et  al.,  2001). 

Underlying  these  issues  is  the  fact  that  evaluation  or  research  expertise  is 
uneven  across  the  different  projects  and  across  County  departments  and  lead 
agencies,  something  the  Evaluation  Design  Panel  called  attention  to.  Some 
County  departments  have  in-house  research  and  evaluation  units,  whereas  other 
departments  do  not.  Project  coordinators  and  program  staff  vary  in  the  amount 
of  education  or  formal  training  they  may  have  had  in  research  and  in  their 
experience  in  conducting  evaluations.  This  variation  has  posed  a  challenge  both 
in  terms  of  developing  the  projects’  evaluations  and  in  recognizing  the  amount  of 
technical  support  individual  projects  may  require.  This  variation  also  potentially 
has  implications  for  how  well  project  evaluations  may  be  carried  out. 

Finally,  several  interviewees  noted  that  the  lead  agencies  were  ambivalent 
about  the  value  of  the  evaluation  process  itself.  This  is  certainly  not  unique  to 
the  LTFSS  Plan.  Research  on  results-based  decision  making  models  and 
budgeting  has  found  that  one  of  the  difficulties  of  implementing  such  models  is 
overcoming  staff  and/or  management  fears  about  assigning  blame  if  the  results 
are  not  positive  (Liner  et  al.,  2001).  Of  course,  the  LTFSS  Plan’s  intent  in 
adopting  the  RBDM  Framework  was  to  focus  on  measurable  performance  and 
accountability.  With  time,  these  fears  may  ease  as  comfort  levels  rise  with 
familiarity;  however,  easing  these  fears  also  points  to  the  need  for  ongoing 
education  about  the  RBDM  Framework  and  its  purpose  at  all  levels  of 
organization,  from  management  to  line  staff. 
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Projects  Expressed  Concern  About  the  Ability  of  the  Evaluation  to  Measure 
Program  Impact 

Interviewees  also  noted  concerns  that  the  Evaluation  Design  may  not  allow 
the  lead  County  agencies  ultimately  to  “prove  effectiveness”  or  to  measure 
“program  impact.”  In  measuring  program  effects  and  linking  interventions  to 
outcomes,  a  number  of  factors  must  be  considered  that  may  affect  one’s  ability 
to  measure  causation.  These  factors  include:  (1)  changes  in  the  economic 
environment  that  may  positively  or  negatively  impact  the  outcome  indicators  of 
interest;  (2)  potential  biases  in  the  selection  of  program  participants;  (3)  effects 
of  other  social  programs  on  the  outcome  indicators  of  interest.  The  difficulty  of 
properly  considering  such  factors  makes  it  difficult  to  identify  the  impact  of  a 
particular  program  (Rossi  et  al.,  1999).  Recognizing  this,  interviewees 
commented  that  they  thought  the  way  a  number  of  LTFSS  projects  were 
currently  designed  would  make  it  impossible  to  evaluate  their  impact.  For 
example,  projects  note,  as  we  discussed  above,  that  incorporating  economic 
conditions  and  other  factors  into  the  forecast  of  future  trends  is  difficult. 

The  RBDM  Framework  acknowledges  that  evaluation  is  difficult.  It  warns 
against  allowing  the  technical  statistics  to  dominate  the  process  but  notes  that  “it 
can  often  help  to  have  a  statistical  expert  as  part  of  the  team,”  because  the  hard 
part  about  the  baselines  is  the  forecasting.  “Turning  the  curve”  analysis- 
changing  an  indicator’s  trend  trajectory— is  an  important  part  of  judging  whether 
a  program  has  been  successful. 

The  example  provided  in  Friedman  (Friedman,  2001,  Section  2.11) 
forecasts  the  number  of  vacant  houses  in  a  community,  which  was  the  indicator 
for  the  result  “stable  community  with  adequate  housing.”  The  forecast  is 
generated  by  combining  data  on  the  trend  in  vacancies  in  the  past,  trends  in 
local  demographic  change,  and  trends  in  economic  conditions.  Program 
success  is  judged  by  its  ability  to  change  the  trend  (e.g.,  in  vacancies)  against 
what  that  trend  would  have  been  in  the  absence  of  the  program  (e.g.,  the 
forecast). 

A  key  element,  then,  is  to  correctly  forecast  future  trends.  Consider,  for 
example,  what  the  trend  in  vacancies  would  look  like  as  the  community  entered  a 
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recession:  Vacancies  would  most  likely  rise  as  residents  moved  out  and  new 
ones  stopped  moving  in.  Thus,  a  program  that  prevented  the  number  of 
vacancies  from  rising  could  be  considered  a  success  in  a  recession  even  if  the 
number  of  vacancies  remained  constant,  the  program  would  be  a  success. 
However,  if  the  forecast  had  not  incorporated  the  effect  of  the  economy  on 
vacancies  and  instead  had  projected  a  flat  or  declining  trend  in  vacancies,  then 
the  project  would  be  judged  a  failure. 

Programs  in  the  LTFSS  Plan  face  exactly  this  challenge.  It  now  appears 
that  the  economic  expansion  has  ended  and  that  the  County  is  entering  into  a 
recession  of  uncertain  length  and  depth.  It  seems  likely  that  such  a  recession 
will  leave  families  with  fewer  resources  and  greater  stress,  which  will  appear  in 
the  data  as  downturns  in  many  of  the  outcomes  of  interest.  A  simple  pre/post 
evaluation  (i.e.,  comparing  outcomes  after  the  intervention  to  outcomes  before 
the  intervention)  or  a  comparison  of  outcomes  with  a  forecast  trendline  that  did 
not  incorporate  the  effect  of  the  economy  will  not  incorporate  such  a  downturn. 
Such  methods  would,  therefore,  conclude  that  the  LTFSS  Plan  had  had  no  effect 
(or  had  made  things  worse). 

In  some  interviews,  it  was  difficult  to  tell  if  the  underlying  issue  was 
confusion  about  the  RBDM  Framework  itself,  methodological  concerns,  or 
resistance  to  the  idea  of  evaluation.  For  example,  project  interviewees 
questioned  whether  other  evaluation  designs  might  be  better  suited  for  individual 
projects  and  felt  that  there  should  be  flexibility  in  projects’  selecting  what 
approach  to  use.  Half  of  project  interviewees,  for  example,  felt  the  RBDM 
Framework  was  too  rigid  because  the  results  had  to  fit  into  the  four  quadrants  of 
the  model  and,  thus,  did  not  allow  for  measurement  of  other  important 
information. 

The  RBDM  Framework  in  fact  anticipates  some  resistance.  As  it  notes,  the 
B[t]he  truth  of  the  matter  is  that  it  is  very  rare  to  find  an  organization  that  ‘wants’ 
to  do  performance  measurement.  The  reasons  for  this  can  range  from 
organization  inertia  to  fear  about  losing  jobs,  and  everything  in  between” 
(Friedman,  2001 ,  Section  3.6).  Friedman  then  specifies  steps  that  organizations 
may  take  to  address  this,  such  as  assigning  a  coach  or  more  training  to  those 
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who  are  resisting  its  use  or  demonstrating  how  the  results  of  evaluation  are 
practical  and  are  used  to  affect  decision-making. 

That  it  can  be  difficult  to  distinguish  confusion,  concern,  or  resistance  is 
evident  in  project  evaluation  deliverables  that  have  identified  (as  required)  a 
number  of  internal  and  external  mediating  factors  that  may  affect  the  ability  of  a 
lead  agency  to  measure  a  project’s  impact.  These  include  the  “buy-in”  of  the 
community;  constraints  imposed  by  shortages  of  services  not  in  the  projects’ 
control,  such  as  housing  or  child  care;  the  willingness  of  participants  to  complete 
training  or  programs;  and  uneven  knowledge  across  County  departments’  line 
staff  about  what  programs  are  available,  limiting  referrals  to  needed  services. 
These  factors  certainly  can  affect  the  ability  of  projects  to  achieve  their  goals,  but 
some — like  community  buy-in  or  participants’  willingness  to  complete  a 
program — are  within  projects’  ability  to  influence,  and  projects  should  be 
expected  to  do  so.  The  RBDM  Framework  notes  that  mediating  factors  in  some 
circumstances  can  serve  as  a  crutch: 

[Tjhe  point  is  that  all  programs’  performance  measures  are  affected 
by  many  factors  beyond  the  particular  program’s  control.  This  lack  of 
control  is  usually  used  as  an  excuse  for  not  doing  performance 
measurement  at  all.  Turnover  rate,  staff  morale,  you  name  it  is 
“beyond  my  control.”  In  fact,  the  more  important  the  performance 
measure  ...  the  less  control  the  program  has  over  it.  This  is  a 
paradox  at  the  heart  of  doing  performance  measurement  well.  If 
control  were  the  overriding  criteria  for  performance  measures  then 
there  would  be  no  performance  measures  at  all  (Friedman,  2001, 
Section  3.1). 

Whatever  its  cause,  some  projects  appear  to  be  frustrated  with  the 
evaluation  component  of  the  Friedman  RBDM  Framework.  As  one  interviewee 
commented: 

Although  it  is  a  useful  planning  tool  at  the  strategic  level,  at  the 
individual  project  level  is  where  it  appears  to  have  been  less  well 
thought  out.  To  answer  the  question:  “Are  you  better  off  now  then 
you  would  have  been?”  is  where  things  break  down,  because  the 
tools  to  fully  address  that  question  have  been  less  well  developed. 
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As  the  RBDM  Framework  indicates,  technical  assistance  and  additional 
training  may  help  clarify  areas  where  projects  are  unclear  about  the  purpose  of 
the  evaluation  and  about  how  to  carry  it  out. 

DISCUSSION 

At  present,  the  RBDM  Framework  seems  to  be  useful  in  promoting  the 
concept  of  performance  accountability;  however,  the  utility  of  the  RBDM 
Framework  as  an  evaluation  framework  remains  unclear.  Initial  experiences  by 
the  lead  County  agencies  with  developing  evaluation  formative  deliverables  and 
in  trying  to  apply  the  RBDM  Framework  raise  several  issues  that  should  be 
monitored  as  the  evaluation  unfolds. 

The  LTFSS  Plan  Evaluation  Design,  which  is  based  on  the  RBDM 
Framework,  has  helped  to  introduce  County  lead  agency  staff  to  the  concept  of 
results-based  accountability  and  performance  measurement.  Specifically,  the 
RBDM  Framework  has  helped  County  departments  and  agencies  to  think  more 
formally  about  program  outcomes,  to  focus  on  a  common  set  of  outcomes,  and 
to  prioritize  departmental  resources.  However,  projects  have  experienced 
various  difficulties  in  applying  the  official  LTFSS  Plan  Evaluation  Design  which  is 
adapted  from  the  RBDM  Framework.  Specifically,  projects  are  having  difficulty 
applying  the  evaluation  component  of  the  RBDM  Framework  in  such  areas  as 
developing  logic  models,  understanding  what  quadrant  a  performance  measure 
may  fall  in,  and  in  identifying  data  sources  and  data  on  historical  or  comparison 
groups  as  called  for  by  the  Evaluation  Design  to  accurately  measure 
performance  measures  and  track  service  delivery.  In  addition,  some  project  staff 
have  raised  methodological  concerns  about  the  Evaluation  Design’s  ability  to 
measure  program  effects  and  to  link  interventions  to  outcomes.  Although  the 
RBDM  Framework  anticipates  some  resistance  to  implementing  the  evaluation 
component  and  acknowledges  that  data  can  be  scarce,  whether  technical 
assistance  and  additional  training  can  fully  address  all  of  these  concerns  remains 
to  be  seen.  As  projects  proceed  with  their  evaluations,  the  utility  of  the  RBDM 
Framework  as  an  evaluation  framework  should  be  monitored. 
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6.  LESSONS  LEARNED  AND  FUTURE  DIRECTIONS 


INTRODUCTION 

This  report  is  designed  to  “evaluate  the  utility  of  the  service  delivery  and 
planning  approach  taken  by  the  LTFSS  Plan  as  a  way  of  organizing  diverse 
efforts  toward  a  general  desired  result  and  in  its  use  of  evaluation  as  a  means  to 
improve  service  delivery.”18  The  development  and  evaluation  of  the  LTFSS  Plan 
were  strongly  influenced  by  the  RBDM  Framework.  As  the  County  moves  to 
expand  the  application  of  the  RBDM  Framework  to  other  parts  of  the  County 
budget,  it  is  worthwhile  to  consider  early  experiences  applying  the  RBDM 
Framework  in  the  context  of  the  LTFSS  Plan  and  to  identify  important  insights 
gained  from  this  experience. 

Following  a  brief  overview,  this  chapter  provides  a  summary  of  the  key 
lessons  learned  based  on  the  experiences  and  suggestions  of  the  LTFSS  Plan 
participants  and  on  RAND’s  analysis  about  the  utility  of  the  RBDM  Framework  in 
the  context  of  the  LTFSS  Plan.  We  also  examine  future  directions  in  applying 
the  RBDM  Framework,  both  in  new  areas  within  the  County  and  in  its  continued 
use  as  part  of  the  LTFSS  Plan  in  the  coming  year. 

OVERVIEW 

The  planning  process  that  led  to  the  development  of  the  LTFSS  Plan  was 
viewed  by  participants,  in  several  respects,  as  representing  a  new  approach  for 
the  County.  First,  the  flowing  of  funds  from  DPSS  to  other  County  departments 
was  seen  as  a  new  and  important  way  for  the  County  to  do  business,  an 
important  step  away  from  traditional  stove-piping  of  departmental  resources. 
The  LTFSS  planning  process  was  also  viewed  as  the  first  Countywide  test  of 
applying  the  RBDM  Framework  to  planning  for  health  and  human  services 
delivery.  Further,  many  participants  commented  that  the  LTFSS  planning 


18Contract  for  the  Evaluation  of  the  Long-Term  Family  Self-Sufficiency 
Plan,  Exhibit  A,  section  2.8,  page  36. 
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process  represented  a  major  shift  toward  broad  community  involvement  and 
toward  a  consensus-building,  collaborative  approach  to  planning. 

As  of  October  2001 ,  two  and  a  half  years  after  the  initiation  of  the  planning 
process  and  two  years  after  the  approval  of  the  Plan,  startup  of  sen/ice  delivery 
to  clients  is  getting  under  way  for  a  number  of  projects.  Few  projects  expended 
anything  approaching  their  budgeted  allocations  in  the  FY  2000-2001  and  the 
early  results  from  FY  2001-2002  do  not  suggest  major  changes  in  rates  of 
expenditures.  Only  about  half  the  projects  have  approved  implementation  plans 
(such  that  they  could  begin  to  serve  clients),  and  even  fewer  projects  have 
approved  evaluation  plans.  The  LTFSS  Plan  will  not  start  to  affect  the  lives  of 
the  County’s  children  and  families  until  services  are  being  provided  in  volume. 
For  most  projects,  the  provision  of  these  services  appears  to  be  at  least  several 
months  away. 

Second,  in  as  much  as  they  use  the  RBDM  Framework,  LTFSS  projects 
are  not  standard  new  projects  or  simply  expansions  of  existing  programs:  The 
fundamental  ideas  of  the  LTFSS  Plan  and  the  RBDM  Framework  included 
increased  community  involvement,  flowing  of  funds  from  the  department 
nominally  receiving  them  to  the  departments  and  agencies  that  can  best  provide 
the  needed  service(s),  and  integration  of  projects  and  services  at  the  planning 
stage  and  at  the  street-level/service  delivery  stage.  Each  of  these  fundamental 
jdeas — increasing  community  involvement,  flowing  funds  between  departments, 
and  integrating  the  service  delivery  system — requires  that  a  number  of  details  be 
worked  out  about  cross-agency  cooperation  and  coordination  and  collaboration 
with  the  community.  Addressing  such  details  would  be  expected  to  stretch  out 
the  initial  implementation  phase. 

Finally,  because  the  LTFSS  Plan  was  a  prototype  for  a  new  model  of 
service  provision  in  the  County,  there  were  few  precedents.  Thus,  even  before 
the  specific  issues  raised  by  a  particular  project  could  be  addressed,  the  general 
issues  involved  in  implementing  the  new  RBDM  Framework  needed  to  be 
identified  and  procedures  developed  to  deal  with  them.  These  tasks  of 
identifying  issues  and  developing  procedures  to  deal  with  them  added  even  more 
time  to  the  initial  implementation  process.  Therefore,  it  seems  plausible  to  argue 
that  the  slow  roll-out  of  the  LTFSS  Plan  projects  is  partially  the  result  of  the 
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application  of  the  RBDM  Framework  and  the  County’s  underlying  objective  of 
changing  the  way  it  does  business  in  providing  services  to  children  and  their 
families.  Some  of  the  delays  are  intrinsic  to  any  application  of  the  model,  and 
some  of  the  delays  are  one-time  events  involved  in  working  out  procedures  to 
deal  with  Friedman-type  projects. 

Despite  the  slow  project  roll-out,  much  progress  has  been  made  and  the 
history  of  the  LTFSS  Plan  to  date  suggests  promise.  The  County  has  shown  that 
it  can  plan  and  implement  according  to  this  new  model — “planning  for  results”  as 
conceptualized  by  the  RBDM  Framework.  At  this  time,  initial  procedures  have 
been  developed  and  most  of  the  steps  have  been  completed  for  at  least  some  of 
the  projects.  As  experience  accumulates,  refined  procedures  and  processes 
should  allow  for  improved  application  of  the  RBDM  Framework. 

Those  involved  in  the  planning  and  implementation  phases  of  the  LTFSS 
Plan  effort  expect  two  types  of  long-term  benefits  from  the  global  strategic 
planning.  First,  interviewees  think  the  LTFSS  Plan  has  stimulated  progress  on 
such  issues  as  allowing  data  sharing  across  agencies  for  common  clients, 
developing  cross-agency  procedures  and  processes  for  reporting  or  contracting, 
improving  working  relationships  between  County  departments  and  agencies, 
establishing  lasting  partnerships  between  the  County  departments  and  the 
community,  and  focusing  County  departments  and  key  partners  on  a  common 
set  of  outcomes  and  on  “planning  for  results”  in  the  development  of  programs. 

Second,  interviewees  see  a  number  of  potential  benefits  to  clients  when  all 
the  LTFSS  projects  have  begun  providing  services.  Projected  benefits  include 
more  services  to  families,  less  fragmentation  of  services  to  these  families, 
improvements  in  customer  service,  more  knowledgeable  line  staff  about  the 
range  of  County  programs  available  to  their  clients,  and  more  effective  referrals. 
More  globally,  interviewees  have  detected  a  positive  shift  toward  a 
comprehensive  approach  to  service  delivery  that  focuses  on  building  up  families’ 
strengths  rather  than  focusing  simply  on  what’s  wrong  or  on  service  delivery  to 
individuals  apart  from  their  family  context. 
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KEY  LESSONS  LEARNED 

From  our  observation  of  the  County’s  first  experience  with  using  the  RBDM 
Framework  and  results-based  decision-making  model  to  develop  and  implement 
the  Plan,  we  draw  four  lessons  that  should  improve  the  utility  of  the  RBDM 
Framework  as  it  is  used  more  widely  in  the  County. 

The  RBDM  Framework  Requires  Significant  Time  for  Planning 

The  RBDM  Framework  calls  for  an  iterative  planning  process  that  starts  by 
identifying  the  desired  end  result(s)  for  population  outcomes  and  working 
backwards  to  means  (Friedman,  2001,  Section  1.1).  In  addition,  the  RBDM 
Framework  encourages  broad  community  involvement  in  the  planning  process 
and  the  identification  of  all  relevant  “partners”  from  the  public  and  private  sectors 
who  may  have  a  role  in  affecting  the  end  result.  Further,  Friedman  urges  the 
iterative  planning  process  to  “fit  the  pieces  together”  and  to  consider  how 
proposed  programs  or  approaches  fit  together  into  a  system  of  services  and 
supports,  “not  just  a  loose  confederation  of  good  ideas.”19  Although  Friedman  is 
not  specific  about  the  amount  of  time  required  to  fully  implement  the  RBDM 
Framework,  he  is  specific  about  the  planning  steps  and  processes  to  be  followed 
to  successfully  undertake  such  a  global  approach  to  planning  and  budgeting. 

In  applying  the  RBDM  Framework,  leaders  and  planning  participants 
involved  in  developing  the  LTFSS  Plan  were  working  under  real-life  constraints. 
They  perceived  that  the  large  pool  of  PIF  and  SA  monies  available  to  the  County 
was  at  risk  of  being  lost  if  a  plan  was  not  finalized  quickly.  Given  that  the  County 
was  just  coming  out  of  a  recession,  there  were  competing  pressures  about  how 
these  dollars  might  be  used.  The  Board  decided  it  wanted  a  comprehensive, 
long-term  plan  developed  for  using  the  estimated  available  PIF  and  SA  monies 
before  these  dollars  were  spent  for  other  purposes.  The  Board  directed  the 
NDTF  to  report  back  to  them  in  six  months  with  a  comprehensive  plan  for 
spending  these  funds  and  other  CalWORKs  dollars  to  achieve  the  desired  result 


19The  Results  and  Performance  Accountability  Implementation  Guide, 
section  2.12,  “How  Do  We  Identify  What  Works  to  Improve  Conditions  of  Well- 
Being?,”  http://www.raauide.Org/2  12.htm. 
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of  helping  families  achieve  long-term  self-sufficiency.  Thus,  the  planners’  charge 
was  to  do  what  could  be  done  within  the  time  frame  allocated  for  this  planning 
process. 

Much  was  accomplished  in  that  short  planning  time  frame.  Through  a 
collaborative,  consensus-building  process,  those  involved  in  the  LTFSS  planning 
process  agreed  upon  a  definition  of  long-term  family  self-sufficiency,  selected  a 
set  of  Countywide  outcome  indicators  to  measure  progress  toward  achieving  that 
goal,  and  developed  a  suite  of  projects  as  a  means  of  achieving  those  goals. 

That  said,  most  LTFSS  planning  leaders  and  participants  would  have  liked 
more  time  to  work  through  the  RBDM  Framework  and  to  fully  consider  the 
different  issues  that  arose.  A  key  lesson  learned  is  that  future  planning  efforts 
using  the  RBDM  Framework  would  benefit  from  more  time  to  work  through  the 
RBDM  Framework’s  steps.  More  time  would  allow  several  benefits.  First,  it 
would  allow  planners  to  bring  the  “best  available  evidence”  to  the  table  in 
determining  what  works  and  what  outcome  indicators  should  be  used  to  measure 
progress  toward  achieving  the  agreed  upon  end  result(s)  or  outcome(s). 
Second,  additional  time  would  also  allow  planners  to  more  fully  involve  all  the 
relevant  stakeholders  (e.g.,  community  advocacy  groups,  researchers,  and 
service  providers)  in  the  planning  process  and  to  assess  what  services  already 
exist  in  the  community  and  how  the  developed  projects  will  relate  to  one  another 
and  to  the  existing  service  delivery  systems.  Finally,  consensus-building  is  hard 
to  do  and  takes  time  to  fully  consider  differing  viewpoints  and  to  reach 
agreement  on  an  approach;  having  more  time  would  allow  planners  to  do  an 
even  more  thorough  job  of  reaching  consensus. 

In  Large  Counties,  More  Formal  Links  Between  Planning  and 
Implementation  Are  Needed 

In  a  large  county  such  as  Los  Angeles  County,  responsibility  for  planning 
and  implementing  often  fall  to  different  people.  The  RBDM  Framework 
emphasizes  community  involvement  at  planning,  but  day-to-day  implementation 
is  the  responsibility  of  County  employees.  This  leaves  room  for  differences  in 
interpretations  of  the  vision  laid  forth  by  the  planners  and  the  possibility  of 
developing  programs  that  may  not  be  feasible  or  represent  the  best  use  of 
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agency  resources.  Further,  planning  and  implementing  an  initiative  the 
magnitude  and  complexity  of  the  LTFSS  Plan  requires  addressing  three  sets  of 
issues.  First,  it  is  essential  to  get  the  high-level  involvement  of  the  relevant 
players  and  of  community  representatives  in  the  planning  process  and 
consensus-building  to  reach  an  agreed-upon  global  action  plan.  Second,  as  the 
LTFSS  Plan  moves  from  planning  to  implementation,  a  number  of  details  must 
be  worked  out  about  cross-agency  cooperation  and  coordination  (e.g.,  the 
flowing  of  funds  across  departments,  standardizing  reporting  procedures  or  MOU 
formats,  determining  what  contracting  requirements  will  be,  and  cross¬ 
supervision  of  departmental  staff).  Third,  the  individual  lead  agency  and 
program  staff  responsible  for  implementing  a  specific  project  must  determine 
what  program  infrastructure  (e.g.,  site  selection,  hiring  and  training  of  staff, 
recruitment  of  program  participants)  must  be  in  place  to  start  up  service  delivery. 
To  successfully  address  these  three  issues,  it  would  have  helped  if  more  formal 
links  between  planning  and  implementation  has  been  specified  (e.g., 
mechanisms  for  ensuring  inclusion  in  the  planning  process  of  County  and  non- 
County  service  providers  or  agency  representatives  who  would  be  responsible 
for  implementing  an  initiative). 

In  practice,  the  LTFSS  Plan  represents  a  change  from  “business  as  usual” 
in  Los  Angeles  County  and  the  innovations  of  the  RBDM  Framework  pose  some 
inherent  challenges  in  addressing  these  three  issues.  The  RBDM  Framework 
encourages  broad  community  input  at  planning  and  emphasizes  collaboration 
and  partnerships,  but  it  is  silent  about  how  this  might  be  accomplished  in  the 
implementation  phase.  The  LTFSS  planners  also  adopted  integration  of  service 
delivery  as  one  of  the  Plan’s  strategies.  However,  actual  implementation  is  a 
full-time  job,  of  necessity,  done  by  lead  agency  staff — employees  of  a  particular 
County  department  or  agency.  Those  involved  in  developing  the  LTFSS  projects 
are  not  necessarily  the  same  individuals  responsible  for  implementing  those 
projects.  Thus,  one  of  the  challenges  is  maintaining  community  involvement  and 
ensuring  that  the  final  projects  faithfully  implement  the  planners’  and 
community’s  intent. 

To  ensure  continuity  of  the  vision  from  the  planning  process  to  the 
implementation  phase  and  to  maintain  the  focus  on  outcomes,  several 
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participants  suggested  that  ongoing  Workgroup  meetings  that  involve  County 
and  non-County  representatives  who  participated  in  developing  the  LTFSS  Plan 
would  be  useful.  Such  meetings  might  be  a  useful  forum  for  addressing 
programmatic  issues  and  constraints  that  planners  may  not  have  fully  considered 
and  for  reconciling  them  with  the  planners’  vision.  Meeting  invitees  might  include 
Workgroup  planning  participants,  lead  agency  and  program  representatives, 
researchers  or  outside  program  experts,  and  contract  and  finance 
representatives. 

Ongoing  technical  assistance  could  be  useful  in  other  areas.  Because  the 
dollars  for  LTFSS  projects  come  out  of  welfare  reform  and  federal  TANF 
legislation,  the  flow  of  these  funds  from  one  department  to  other  receiving 
departments  or  agencies  complicates  implementation.  For  example,  senior 
planning  leaders  felt  that  finance  and  contracts  representatives  should  have 
been  involved  earlier  in  the  planning  phase.  Doing  so  would  have  allowed  them 
an  earlier  start  at  addressing  such  issues  as  cross-agency  flow  of  funding,  and 
setting  up  agreed-upon  reporting  procedures  and  document  templates  in 
preparation  for  the  implementation  phase.  Also,  interviewees  recognize  that 
ongoing  technical  assistance  is  valuable  to  project  staff  in  the  receiving 
departments  or  agencies  to  help  them  develop  project  implementation  plans  and 
contracts  that  meet  CDSS  and  federal  TANF  requirements. 

Planning  and  Coordination  Require  Considerable  Resources 

Another  lesson  learned  was  the  importance  of  recognizing  the  resource 
demands  of  the  numerous  coordination  activities  that  must  occur  early  in  the 
planning  phase  and  continuing  through  the  early  part  of  the  implementation 
phase.  As  noted  by  several  senior  management  leaders,  the  LTFSS  Plan’s 
budget  divides  a  project’s  funds  evenly  across  the  five  years  and,  by  doing  so, 
implicitly  assumes  rapid  program  start-up  and  scale-up  to  full  operating  capacity. 
However,  as  discussed  above,  to  make  the  changes  “from  business  as  usual” 
embodied  in  the  LTFSS  Plan  and  to  undertake  the  innovations  of  the  RBDM 
Framework,  a  number  of  issues  related  to  cross-agency  cooperation  and 
coordination  must  be  addressed  early  on.  In  addition,  a  great  deal  must  occur 
before  projects  are  ready  for  sen/ice  delivery.  In  particular,  projects  must 
contract  with  service  providers,  select  program  sites,  hire  and  train  staff,  recruit 
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program  participants,  develop  program  materials,  among  a  host  of  other 
logistical  issues.  Further,  as  one  senior  management  leader  noted,  it  can  be  a 
cumbersome  process  to  spend  dollars  in  the  County,  with  such  tasks  as  putting 
contracts  into  place  taking  between  6  and  8  months,  obtaining  facility  space 
taking  a  minimum  of  12  months,  and  hiring  staff  taking  at  least  6  months. 
Interviewees  felt  that  budgets  should  realistically  reflect  the  life  cycle  of  a 
project — where  one  would  expect  to  see  a  period  of  ramping  up,  full-scale 
operations,  and  then  ramping  down — but  that  budgets  should  also  build  in  early 
funding  to  address  the  numerous  coordination  issues  between  agencies  prior  to 
service  delivery. 

The  Evaluation  Component  Is  More  Complicated  Than  It  Appears  at  First 

Evaluation  plays  a  prominent  role  in  the  LTFSS  Plan.  An  outside 
evaluator — RAND — is  responsible  for  conducting  a  Countywide  evaluation  of  the 
process  of  developing,  implementing,  and  evaluating  the  Plan  and  of  the  impact 
of  the  LTFSS  Plan  on  County  operations  and  on  the  Plan’s  designated 
outcomes,  such  as  infant  mortality  and  poverty  rates.  Also,  each  of  the  LTFSS 
projects  is  required  to  develop  an  evaluation  report  in  accordance  with  the 
Evaluation  Design  to  assess  program  impact.  Within  the  RBDM  Framework’s 
results  and  performance  accountability  method,  evaluation  is  key  to  focusing  the 
County’s  efforts  on  the  set  of  26  LTFSS  Plan  outcome  indicators  and  to 
measuring  progress  toward  the  LTFSS  Plan’s  goal  of  helping  families  to  achieve 
long-term  self-sufficiency. 

Consistent  with  the  RBDM  Framework,  the  LTFSS  Plan’s  Evaluation 
Design  emphasizes  that  progress  on  selected  performance  measures  must  be 
measured.  Among  the  fundamental  insights  of  the  RBDM  Framework  is  the 
importance  of  refocusing  project  efforts  away  from  a  narrow  focus  on  what  the 
project  did  to  a  broader  focus  on  how  the  project  affected  client-level  outcomes 
and  well-being.  To  that  end,  the  LTFSS  Evaluation  Design  requires  that  each 
project  explicitly  state  which  outcomes  it  expects  to  affect,  operationalizing  those 
outcomes  through  specific  measures  and  then  carefully  tracking  the  evolution  of 
those  measures  over  time.  Individual  projects  are  urged  to  track  whether  its 
“customers  are  better  off”  (Friedman,  2001 ,  Section  3.16). 
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Building  an  evaluation  component  into  the  LTFSS  Plan  helps  to  keep  the 
key  stakeholders  and  lead  agencies  focused  on  outcomes  and  informs  the  Board 
about  how  these  funds  were  used.  The  Plan’s  evaluation  component  also 
introduces  County  departmental  staff  to  Friedman’s  approach  and  the  notion  of 
explicitly  thinking  about  what  a  program  is  trying  to  achieve  and  how  to  assess 
whether  what  an  agency  is  doing  is  having  the  desired  effects.  Participants  in 
the  planning  and  implementation  of  the  LTFSS  Plan  have  praised  the  application 
of  the  RBDM  Framework  for  having  stimulated  and  guided  County  departments 
and  other  lead  agencies  in  thinking  more  formally  about  program  outcomes  and 
as  providing  a  useful  framework  for  prioritizing  departmental  resources. 

However,  some  interviewees  also  expressed  concern  about  whether  the 
Evaluation  Design  will  allow  for  the  determination  of  causation,  i.e.,  the  effect  of 
an  individual  LTFSS  project  (or  even  the  Plan  as  a  whole)  on  outcomes.  The 
RBDM  Framework  is  both  quite  specific  and  vague  about  how  to  evaluate  the 
effect  of  a  given  project  or  a  broad  effort.  The  specific  instructions  involve  using 
historical  information  to  create  a  baseline  (see  Friedman,  2001,  Section  2.11) 
and  then  using  the  historical  trend  information  to  project  a  future  trend  in  the 
absence  of  new  programs.  Program  success  is  evaluated  in  terms  of  deviations 
between  the  projected  trend  line  and  actual  outcome.  If  historical  data  are  not 
available,  the  RBDM  Framework  directs  projects  to  collect  data  on  comparison 
groups. 

However,  the  RBDM  Framework  notes  that  simple  pre-  versus  post¬ 
comparisons  are  invalid  because  other  non-project  forces  would  cause  outcomes 
to  change  even  in  the  absence  of  a  new  program.  The  examples  in  the  RBDM 
Framework  and  the  description  of  the  “Story  Behind  the  Baseline”  process 
emphasize  that  many  factors  will  affect  future  outcomes  (see  the  example  about 
“adequate  housing”  in  Section  2.11  of  Friedman,  2001)  and  that  not  all  the 
factors  are  captured  by  a  simple  trendline.  Still,  neither  the  RBDM  Framework 
nor  the  CAO’s  technical  assistance  materials  offers  specific  guidance  about  how 
to  control  for  such  other  factors  in  the  historical  projections  or  the  comparison 
group  designs. 

Those  performing  project  evaluations  would  benefit  from  additional  training 
and  technical  assistance.  Current  CAO  training  materials  describe  the 
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application  of  the  basic  trendline  methods  suggested  by  Friedman.  However,  the 
recent  path  of  the  economy — rapid  expansion,  followed  by  at  least  a  mild 
recession — suggests  that  trendline  methods  are  likely  to  underestimate  the  true 
effects  of  the  program.  The  trendline — computed  during  the  expansion — 
implicitly  assumes  that  the  expansion  will  continue.  Applying  trendline  methods, 
any  deviation  from  that  trendline — e.g.,  because  of  the  end  of  the  expansion — 
will  be  attributed  to  the  program.  Thus,  for  some  outcomes,  simple  trendline 
analyses  are  likely  to  suggest  that  the  Plan  made  outcomes  worse.  There  exist 
alternative  methodologies  that  will  (at  least  partially)  correct  for  economic 
conditions.  Thus,  it  may  be  useful  to  consider  significant  increases  in  the  level  of 
(and  funding  for)  training  for  those  performing  the  project  evaluations  and 
technical  assistance  in  developing,  refining,  and  implementing  their  evaluation 
plans  so  that  they  can  detect  any  positive  effects  of  the  Plan,  despite  a  slowing 
of  economic  growth. 

Finally,  interviewee  comments  suggest  that  while  the  RBDM  Framework  is 
accepted  as  a  useful  guide  to  planning,  there  may  be  some  resistance  to  it  in 
practice  (i.e.,  in  evaluating  how  successful  projects  are  in  affecting  certain 
outcomes  and  using  this  information  to  guide  future  decision-making  and 
funding).  The  RBDM  Framework,  in  fact,  anticipates  this  and  specifies  steps  that 
organizations  may  take  to  address  such  resistance.  Those  steps  largely  entail 
ongoing  education  and  training  in  the  principles  of  the  RBDM  Framework  and 
how  it  can  be  valuable  for  decision-making. 

FUTURE  DIRECTIONS 

These  lessons  learned  become  a  useful  base  on  which  to  build  as  the 
County  moves  forward  to  both  extend  the  application  of  the  RBDM  Framework 
outside  of  the  LTFSS  Plan  and  to  continue  the  use  of  the  RBDM  Framework 
within  it. 

LTFSS  Plan  Has  Laid  the  Groundwork  for  Extending  the  Application  of  the 
RBDM  Framework 

The  LTFSS  Plan  represents  a  new  model  of  service  provision  in  the  County 
based  on  broad  community  involvement  in  planning  and  in  determining  “what 
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works”  to  achieve  desired  results.  This  new  model  of  service  provision  also 
directly  confronts  the  challenges  inherent  in  overcoming  bureaucratic  barriers  to 
improve  cross-agency  coordination  and  collaboration  in  service  delivery  and  in 
working  together  with  the  community  to  comprehensively  meet  families’  needs. 

A  number  of  key  stakeholders  also  view  the  LTFSS  Plan  as  a  significant 
application  of  the  RBDM  Framework  in  the  County.  The  County’s  experience  in 
the  LTFSS  Plan’s  development  and  implementation  is  seen  as  an  important 
opportunity  to  learn  what  adjustments  may  be  needed  in  applying  the  results- 
based  decision-making  model  to  other  areas  of  the  County’s  budget.  Such 
learning  opportunities  are  particularly  important  given  the  County’s  recent 
decision  to  incorporate  the  RBDM  Framework  and  results-based  decision¬ 
making  model  into  the  County’s  budget  process,  specifically,  into  the  Children’s 
Budget  to  enable  the  County  to  measure  progress  toward  Goal  5,  Strategy  2  in 
five  outcome  areas.  Phased  implementation  of  this  plan  has  begun,  with  County 
departments  initially  being  requested  to  help  the  CAO  develop  an  inventory  of 
current  County  programs  that  serve  children  and  families.  The  development  of 
program  performance  measures  by  County  departments  using  the  RBDM 
Framework  is  slated  to  begin  in  FY  2002-2003,  with  full  implementation 
projected  by  FY  2006-2007  (CAO/SIB,  2001). 

In  its  use  of  the  RBDM  Framework  in  the  LTFSS  Plan  and  in  its  extension 
to  other  parts  of  the  County  budget,  the  County  is  ahead  of  other  jurisdictions  in 
applying  this  RBDM  Framework  to  best  serve  the  needs  of  children  and  low- 
income  families.  Further,  by  incorporating  an  evaluation  component  into  the 
Plan,  the  County  is  ahead  of  others  in  its  quest  to  leam  and  improve  on  the 
RBDM  Framework’s  application. 

The  Continued  Application  of  the  RBDM  Framework  Within  LTFSS  Plan 
Presents  Opportunities  and  Challenges 

As  the  LTFSS  Plan  moves  into  its  third  calendar  year  since  approval  of  the 
Plan,  lead  agencies  and  LTFSS  projects  enter  a  new  phase.  From  a 
management  perspective,  lead  agencies  will  move  from  an  emphasis  on 
developing  projects’  implementation  plans  and  putting  an  initial  program 
infrastructure  in  place  to  an  emphasis  on  service  delivery,  refining  LTFSS 
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projects,  overseeing  contractors,  and  evaluating  these  projects  and  tracking 
outcomes.  According  to  the  RBDM  Framework,  the  process  is  iterative.  We  are 
now  well  into  the  first  cycle  of  planning  (project  selection  and  funding  decisions), 
implementation,  and  evaluation.  Lessons  learned  from  implementation  and 
evaluation  should  then  cycle  back  into  a  follow-on  planning  phase. 

Based  on  those  lessons,  some  projects  would  have  their  funding 
increased,  some  projects  would  have  their  funding  decreased,  some  projects 
would  be  terminated,  and  some  new  projects  would  be  initiated  based  on  new  or 
newly  perceived  needs  and  new  program  models  developed  elsewhere. 
Moreover,  California  as  a  whole  and  the  County  face  likely  changes  in  their  fiscal 
situation.  It  is  too  early  to  predict  what  changes  will  be  enacted,  but  it  seems 
likely  that  funding  will  be  considerably  tighter  and  that  there  may  be  pressures  to 
use  some  of  the  PIF  monies  for  basic  services. 

Clearly,  according  to  the  RBDM  Framework,  future  LTFSS  planning 
choices  should  be  influenced  by  the  accumulating  evidence.  Successful  and  fast 
implementation  contributes  to  a  case  for  continued  and  perhaps  increased 
funding.  Similarly,  RBDM  Framework-based  evaluation  evidence  of 
effectiveness  should  also  contribute  to  a  case  for  continued  and  perhaps 
increased  funding.  Finally,  conventional  research  evidence  of  program  efficacy 
and  cost-effectiveness  should  also  contribute  to  a  case  for  continued  and 
perhaps  increased  funding.  Conversely,  programs  that  rolled  out  slowly,  had 
poor  RBDM  Framework-based  evaluation  outcomes,  and  had  limited  or  negative 
research  evidence  from  elsewhere  should  be  at  higher  risk  of  lower  funding  or 
even  termination. 

The  changed  operating  philosophy  embodied  in  the  LTFSS  Plan  itself  has 
begun  to  stimulate  real  cultural  change  in  the  County  and  the  lead  agencies. 
Over  the  next  year,  lead  agencies  have  an  opportunity  to  show  that  their 
programs  can  contribute  to  the  well-being  of  these  families  and  can  positively 
impact  the  outcomes  of  interest. 
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